RT Journal Article
SR Electronic
T1 Reproducible comparison and interpretation of machine learning classifiers to predict autism on the ABIDE multimodal dataset
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2024.09.04.24313055
DO 10.1101/2024.09.04.24313055
A1 Dong, Yilan
A1 Batalle, Dafnis
A1 Deprez, Maria
YR 2024
UL http://medrxiv.org/content/early/2024/09/04/2024.09.04.24313055.abstract
AB Autism is a neurodevelopmental condition affecting ∼1% of the population. Recently, machine learning models have been trained to classify participants with autism using their neuroimaging features, though the performance of these models varies in the literature. Differences in experimental setup hamper the direct comparison of different machine-learning approaches. In this paper, five of the most widely used and best-performing machine learning models in the field were trained to classify participants with autism and typically developing (TD) participants, using functional connectivity matrices, structural volumetric measures and phenotypic information from the Autism Brain Imaging Data Exchange (ABIDE) dataset. Their performance was compared under the same evaluation standard. The models implemented included: graph convolutional networks (GCN), edge-variational graph convolutional networks (EV-GCN), fully connected networks (FCN), auto-encoder followed by a fully connected network (AE-FCN) and support vector machine (SVM). Our results show that all models performed similarly, achieving a classification accuracy around 70%. Our results suggest that different inclusion criteria, data modalities and evaluation pipelines rather than different machine learning models may explain variations in accuracy in published literature. The highest accuracy in our framework was obtained by an ensemble of GCN models trained on combination of functional MRI and structural MRI features, reaching classification accuracy of 72.2% and AUC = 0.78 on the test set. The combined structural and functional modalities exhibited higher predictive ability compared to using single modality features alone. Ensemble methods were found to be helpful to improve the performance of the models. Furthermore, we also investigated the stability of features identified by the different machine learning models using the SmoothGrad interpretation method. The FCN model demonstrated the highest stability selecting relevant features contributing to model decision making. Code available at: https://github.com/YilanDong19/Machine-learning-with-ABIDE.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis work was supported by King's -China Scholarship Council (K-CSC) [grant number 202008060109]. The authors acknowledge support in part from the Wellcome Engineering and Physical Sciences Research Council (EPSRC) Centre for Medical Engineering at Kings College London [WT 203148/Z/16/Z], and the NIHR Maudsley Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:ABIDE database: https://fcon_1000.projects.nitrc.org/indi/abide/ Data acquire: http://preprocessed-connectomes-project.org/abide/I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesAll data used in the study are available online at the ABIDE dataset repository http://fcon_1000.projects.nitrc.org/indi/abide/ http://fcon_1000.projects.nitrc.org/indi/abide/ http://preprocessed-connectomes-project.org/abide/