Summary
Background Multiple Sclerosis (MS), is a chronic idiopathic demyelinating disorder of the CNS. Imaging plays a central role in diagnosis and monitoring. Monitoring for progression however, can be repetitive for neuroradiologists, and this has led to interest in automated lesion detection. Simultaneously, in the computer science field of Remote Sensing, Change Detection (CD), the identification of change between co-registered images at different times, has been disrupted by the emergence of Vision Transformers. CD offers an alternative to semantic segmentation leveraging the temporal information in the data.
Methods In this retrospective study with external validation we reframe the clinical radiology task of new lesion identification as a CD problem. Consecutive patients who had MRI studies for MS at our institution between 2019 and 2022 were reviewed and those with new lesion(s) were included. External data was obtained from the MSSEG2 challenge and OpenMS. Multiple CD models, and a novel model (NeUFormer), were trained and tested. Results were analysed on both paired slices and at the patient level. Expected Cost (EC) and F2 were independently and prospectively chosen as our primary evaluation metrics. For external data we report DICE and F1 to allow for comparison with existing data. For each test set 1000 bootstrapping simulations were performed by sampling 10 patient samples with replacement giving a non parametric estimate of the confidence interval. Wilcoxon statistics were calculated to test for significance.
Findings 43,440 MR images were included for analysis (21,720 pairs). The internal set comprised of 170 patients (110 for training, 30 for tuning, 30 testing) with 120 females and 50 males, average age of 42 (range 21 – 74). 60 (40 + 20) patients were included for external validation.
In the CD experiments (2D) our proposed NeuFormer model achieved the best (lowest) Expected Cost (EC) (p=0.0095), the best F2 and second best DICE (p<0.0001). At the patient level our NeUFormer model had the joint highest number of True Positive lesions, and lowest number of False negatives (p<0.002). For CD on external data, NeUFormer achieved the highest DICE on both datasets (p<0.0001). NeUFormer had the lowest or joint lowest number of False Positives on external data (p<0.0001 in all cases).
Interpretation Reformulating new lesion identification as a CD problem allows the use of new techniques and methods of evaluation. We introduce a novel Siamese U-Transformer, NeUFormer, which combines concepts from U-Net, Siamese Networks, and vision transformers to create a model with improved small lesion detection and the consistently best EC. Its ability to increase detection of small lesions, balanced with relatively few false positives, and superior generalisability has the potential to greatly impact the field of the identification of radiologic progression of MS with AI.
Evidence before this study Multiple Sclerosis (MS), a chronic and idiopathic demyelinating disorder of the CNS, is diagnosed using the McDonald criteria based on MRI interpretation. Without a definitive MS biomarker, AI holds promise is for uncovering unique features indicative of MS, improving diagnostics and identifying progression. Research in the field typically centres on segmentation and classification, leaving a gap in evaluating temporal imaging changes. The MSSEG2 challenge has now enabled more research into new lesion identification in MS. Even so, most solutions are based on semantic segmentation architectures and rely on limited metrics for evaluation. The identification of small lesions also remains a challenge.
Remote Sensing (RS) is the science of obtaining information about objects or areas from a distance, typically from aircraft or satellites. In the RS literature, Change Detection (CD) refers to the identification of significant alterations in co-registered images captured at different times. In this way CD offers an alternative to semantic segmentation leveraging the temporal information in the data. This field was dominated by convolutional neural networks but has recently been disrupted by transformer-based architectures. Transformers, fuelled by their success in NLP, are gaining popularity across all computer vision tasks due to their larger effective receptive field and enhanced context modelling between image pixels. Inspired by these developments, we incorporate some of these ideas into our NeUFormer model.
Added value of this study This study redefines the task of identifying progression on MRI brain in MS as a CD problem, borrowing concepts from RS. This approach allows for both pixel- and patient-level evaluation and rethinks standard metrics to suit specific clinical needs. This acknowledges the distinction between trivial variation in segmentation and clinically significant change. State-of-the-art CD models are assessed at this task, and a novel model, NeuFormer, is introduced. NeuFormer synergistically combines concepts from the classical U-Net (which was originally intended for brain segmentation), Siamese architecture adaptations specifically for CD, Swin-UNETR (a U-Transformer developed by MONAI to integrate the shifting window structure of the Swin transformer into medical imaging) and ChangeFormer which also uses attention at scale specifically for CD, leveraging improved spaciotemporal reasoning to create a model which is better for small lesion identification and with the consistently lowest EC associated with its decisions.
Implications of all the available evidence Reframing lesion identification as CD enables an alternative to semantic segmentation leveraging the temporal information in the data, enhancing the model’s relevance and customization for specific medical tasks. We also propose the flexible Expected Cost metric, as it facilitates varying action thresholds and helps to customise tools to stakeholder preferences.
Siamese vision transformers show promise for CD on MRI in MS including for smaller lesions which are traditionally difficult for computer vision models to identify. This may be to the intrinsic spaciotemporal advantages of vision transformers, with positional embedding, over patch based convolutional methods.
NeUFormer’s ability to increase detection of small lesions, balanced with relatively few false positives and excellent generalisability has the potential to greatly impact the field of the identification of radiologic progression of MS with AI.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was performed within the Irish Clinical Academic Training (ICAT) Programme, supported by the Wellcome Trust and the Health Research Board (Grant No. 203930/B/16/Z), the Health Service Executive National Doctors Training and Planning and the Health and Social Care, Research and Development Division, Northern Ireland and the Faculty of Radiologists, Royal College of Surgeons in Ireland. This research was supported by Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289_P2 and by a Fulbright-HRB HealthImpact Scholarship.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Ethics committee/IRB of St Vincent's University Hospital Dublin Ireland gave ethical approval for this work
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors
