Abstract
To develop machine learning methods to quantify joint damage in patients with rheumatoid arthritis (RA), we developed the RA2 DREAM Challenge, a crowdsourced competition that utilized existing radiographic images and “gold-standard” scores on 674 sets of films from 562 patients. Training and leaderboard sets were provided to participants to develop methods to quantify joint space narrowing and erosions. In the final round, participants submitted containerized codes on a test set; algorithms were evaluated using weighted root mean square error (RMSE). In the leaderboard round, there were 173 submissions from 26 teams in 7 countries. Of the 13 submissions in the final round, four top-performing teams were identified. Robustness of results was assessed using Bayes factor and validated using an independent set of radiographs. The top-performing algorithms, which consisted of different styles of deep learning models, provided accurate and robust quantification of joint damage in RA. Ultimately, these methods lay the groundwork to accelerate research and help clinicians to optimize treatments to minimize joint damage.
Competing Interest Statement
The following authors have financial interests in these companies: YG, Genentech, F. Hoffmann-La Roche AG, Lilly, Merck KGaA, Ann Arbor Algorithms Inc; MM, Bristol-Myers Squibb; PG, Oryn Therapeutics, CannBioRx, Gilead; JCC, PrecisionProfile. All other authors have no competing interests to disclose.
Funding Statement
We would like to acknowledge funding support from Bristol Myers Squibb (BMS) and the CTSA Program National Center for Data to Health (CD2H) is supported by the National Center for Advancing Translational Sciences (NCATS) at the National Institutes of Health (Grant U24TR002306).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
the UAB Institutional Review Board for Human Use gave ethical approval for this work
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
All radiogrpahic images used in this study are available upon reasonable request to the authors. Method write-ups and Docker containers for all submitted methods are available from the Challenge workspace (www.doi.org/10.7303/syn20545111).
