Objective: Assess the effect of random train–test splits on performance in a small T1-weighted MRI radiomics dataset with machine learning.

Background: Radiomic MRI analysis is increasingly being investigated for Parkinson’s disease (PD) detection, and has gained interest for its ability to extract quantitative features that capture subtle brain alterations relevant to PD; however, small, high-dimensional datasets often produce unstable and biased ML performance that can overestimate accuracy. Prior ML studies show that small samples and single random splits yield variable and overoptimistic estimates[1]. This study investigates how train–test splits affect classifier stability using PPMI T1-weighted MRIs [2].

Method: T1-weighted MRI scans with identical acquisition parameters, from 144 participants (72 PD, 72 HC, demographically matched) were obtained from PPMI. Subcortical regions were segmented using FastSurfer [3]. Radiomic features were extracted from ten bilateral subcortical regions using PyRadiomics [4], 24 subjects were reserved as a fixed held-out test set. The remaining 120 subjects were used for analysis, and were split into training (n=96) and test (n=24). A leakage free pipeline including normalization, ReliefF plus Lasso feature selection (selecting top 10 features), and logistic regression classification was implemented within GridSearchCV using inner cross-validation on only the training set. This entire process was repeated using 21 different random seeds to generate the train/test split. A second analysis repeated the same pipeline without the feature selection step, utilizing only the eight features most commonly selected across the seeds in the first analysis, to assess feature robustness and model stability.

Results: Across seeds, using all features, average accuracy ranged from 0.46 to 0.87 (avg: 0.64) on the seed-specific test set and 0.42–0.71 (avg: 0.61) on the fixed held-out set. Restricting the model to eight robust features improved accuracy mean and consistency (test set: avg:0.81 (0.67–0.92); fixed held-out test set 0.70 (0.62–0.75)).

Conclusion: Radiomic classification performance in small PD datasets is highly sensitive to random train–test split. Evaluating models across multiple seeds and reporting variability is essential for transparency and reproducibility. Using consistently selected robust features improves model stability and generalizability.

References: [1] An C, Park YW, Ahn SS, Kim H, and Lee SK, “Radiomics machine learning study with a small sample size: Single random training-test set split may lead to unreliable results,” PLoS One, vol. 16, no. 8, p. e0256152, Aug. 2021, doi: e0256152.
[2] “Parkinson’s Progression Markers Initiative (PPMI).” [Online]. Available: https://www.ppmi-info.org/
[3] Henschel L, Conjeti S, Estrada S, Diers K, Fischl B, and Reuter M, “A fast and accurate deep learning based neuroimaging pipeline,” NeuroImage, vol. 219, p. 117012, 2020, doi: 10.1016/j.neuroimage.2020.117012.
[4] van Griethuysen et al., “Computational Radiomics System to Decode the Radiographic Phenotype,” Cancer Res., vol. 77, no. 21, pp. e104–e107, doi: 10.1158/0008-5472.CAN-17-0339.

« Back to 2026 International Congress

MDS Abstracts - https://www.mdsabstracts.org/abstract/effect-of-random-training-test-splits-on-radiomic-classification-of-parkinsons-disease-from-t1-weighted-mri-in-machine-learning/