PT - JOURNAL ARTICLE AU - Traut, Nicolas AU - Heuer, Katja AU - Lemaître, Guillaume AU - Beggiato, Anita AU - Germanaud, David AU - Elmaleh, Monique AU - Bethegnies, Alban AU - Bonnasse-Gahot, Laurent AU - Cai, Weidong AU - Chambon, Stanislas AU - Cliquet, Freddy AU - Ghriss, Ayoub AU - Guigui, Nicolas AU - de Pierrefeu, Amicie AU - Wang, Meng AU - Zantedeschi, Valentina AU - Boucaud, Alexandre AU - van den Bossche, Joris AU - Kegl, Balázs AU - Delorme, Richard AU - Bourgeron, Thomas AU - Toro, Roberto AU - Varoquaux, Gaël TI - Insights from an autism imaging biomarker challenge: promises and threats to biomarker discovery AID - 10.1101/2021.11.24.21266768 DP - 2022 Jan 01 TA - medRxiv PG - 2021.11.24.21266768 4099 - http://medrxiv.org/content/early/2022/03/31/2021.11.24.21266768.short 4100 - http://medrxiv.org/content/early/2022/03/31/2021.11.24.21266768.full AB - MRI has been extensively used to identify anatomical and functional differences in Autism Spectrum Disorder (ASD). Yet, many of these findings have proven difficult to replicate because studies rely on small cohorts and are built on many complex, undisclosed, analytic choices. We conducted an international challenge to predict ASD diagnosis from MRI data, where we provided preprocessed anatomical and functional MRI data from > 2,000 individuals. Evaluation of the predictions was rigorously blinded. 146 challengers submitted prediction algorithms, which were evaluated at the end of the challenge using unseen data and an additional acquisition site. On the best algorithms, we studied the importance of MRI modalities, brain regions, and sample size. We found evidence that MRI could predict ASD diagnosis: the 10 best algorithms reliably predicted diagnosis with AUC∼0.80 – far superior to what can be currently obtained using genotyping data in cohorts 20-times larger. We observed that functional MRI was more important for prediction than anatomical MRI, and that increasing sample size steadily increased prediction accuracy, providing an efficient strategy to improve biomarkers. We also observed that despite a strong incentive to generalise to unseen data, model development on a given dataset faces the risk of overfitting: performing well in cross-validation on the data at hand, but not generalising. Finally, we were able to predict ASD diagnosis on an external sample added after the end of the challenge (EU-AIMS), although with a lower prediction accuracy (AUC=0.72). This indicates that despite being based on a large multisite cohort, our challenge still produced biomarkers fragile in the face of dataset shifts.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis study did not receive any fundingAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Department of Medical Affairs IRB00006966 Institut Pasteur IRB #1 "Genetics and Neurobiology of Susceptibility to psychiatric disorders" gave ethical approval for this workI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesData for the public dataset are available online at https://github.com/neuroanatomy/autism-challenge https://github.com/neuroanatomy/autism-challenge