PT  - JOURNAL ARTICLE
AU  - Mellem, Monika S.
AU  - Kollada, Matt
AU  - Tiller, Jane
AU  - Lauritzen, Thomas
TI  - Explainable AI enables clinical trial patient selection to retrospectively improve treatment effects in schizophrenia
AID  - 10.1101/2021.01.11.20248788
DP  - 2021 Jan 01
TA  - medRxiv
PG  - 2021.01.11.20248788
4099  - http://medrxiv.org/content/early/2021/03/13/2021.01.11.20248788.short
4100  - http://medrxiv.org/content/early/2021/03/13/2021.01.11.20248788.full
AB  - Background Heterogeneity among patients’ responses to treatment is prevalent in psychiatric disorders. Personalized medicine approaches – which involve parsing patients into subgroups better indicated for a particular treatment – could therefore improve patient outcomes and serve as a powerful tool in patient selection within clinical trials. Machine learning approaches can identify patient subgroups but are often not “explainable” due to the use of complex algorithms that do not mirror clinicians’ natural decision-making processes.Methods Here we combine two analytical approaches – Personalized Advantage Index and Bayesian Rule Lists – to identify paliperidone-indicated schizophrenia patients in a way that emphasizes model explainability. We apply these approaches retrospectively to randomized, placebo-controlled clinical trial data to identify a paliperidone-indicated subgroup of schizophrenia patients who demonstrate a larger treatment effect (outcome on treatment superior than on placebo) than that of the full randomized sample as assessed with Cohen’s d. For this study, the outcome corresponded to a reduction in the Positive and Negative Syndrome Scale (PANSS) total score which measures positive (e.g., hallucinations, delusions), negative (e.g., blunted affect, emotional withdrawal), and general psychopathological (e.g., disturbance of volition, uncooperativeness) symptoms in schizophrenia.Results Using our combined explainable AI approach to identify a subgroup more responsive to paliperidone than placebo, the treatment effect increased significantly over that of the full sample (p<0.0001 for a one-sample t-test comparing the full sample Cohen’s d=0.82 and a generated distribution of subgroup Cohen’s d’s with mean d=1.22, std d=0.09). In addition, our modeling approach produces simple logical statements (if-then-else), termed a “rule list”, to ease interpretability for clinicians. A majority of the rule lists generated from cross-validation found two general psychopathology symptoms, disturbance of volition and uncooperativeness, to predict membership in the paliperidone-indicated subgroup.Conclusions These results help to technically validate our explainable AI approach to patient selection for a clinical trial by identifying a subgroup with an improved treatment effect. With these data, the explainable rule lists also suggest that paliperidone may provide an improved therapeutic benefit for the treatment of schizophrenia patients with either of the symptoms of high disturbance of volition or high uncooperativeness.Trial Registration clincialtrials.gov identifier: NCT 00083668; registered May 28, 2004Competing Interest StatementAll authors are current or previous employees of BlackThorn Therapuetics and hold stock or stock options in BlackThorn Therapeutics.Clinical Trialclincialtrials.gov identifier: NCT00083668Funding StatementFunding was provided by BlackThorn Therapeutics whose employees were involved designing this post-hoc analysis, performing the analysis, interpreting the results, and writing the manuscript.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This study makes use of de-identified data made available via the YODA Project (https://yoda.yale.edu/; approved research proposal number 2019-4080) and was exempt from ethical oversight. Regarding the original data collection performed by the investigators of the clinical trial, the study protocol states that it was approved by an Institutional Review Board or an Independent Ethics Committee at each center involved in the trial, but no further details are available.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe data that support the findings of this study are available from the YODA Project (https://yoda.yale.edu/) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. https://yoda.yale.edu/ AbbreviationsAIArtificial IntelligenceAUCArea Under the ROC CurveBRLBayesian Rule ListFDAU.S. Food and Drug AdministrationPANSSPositive and Negative Syndrome ScalePAIPersonalized Advantage IndexPBOPlaceboROCReceiver Operating CharacteristicRest-indRest-indicatedTRTTreatmentTRT-indtreatment-indicatedXAIExplainable Artificial IntelligenceYODAYale University Open Data Access