Longitudinal automated brain volumetry vs. expert visual assessment of atrophy progression on MRI is robust but caution is advised

1. Introduction

In Western countries globally, aging populations anticipate increasing socioeconomic burdens from cognitive and memory decline¹. Structural MRI has been the main imaging modality in the early work-up of cognitive decline^2,3,4,5,6. Recently, numerous automated brain segmentation and volumetric analysis (AVA) tools have emerged to address the shortcomings of subjective expert-based visual assessments (EVA) of brain MRIs and to assist neuro-/radiologists to quicker establish a diagnosis in the early work-up of cognitive decline⁷. Yet, their robustness in longitudinal settings and performance in comparison to neuroradiological experts in real-life follow-up scenarios remain uncertain. Thus, we investigated if expert-based brain atrophy assessment would be reflected by AVA tools in longitudinal routine clinical imaging setting.

The number of patients living with dementia is expected to triple by 2050¹. Hence, the exploration of early biomarkers to predict a conversion to a manifested dementia in possible early stages like subjective cognitive decline⁸ and mild cognitive impairment (MCI)⁹ will be crucial in the future. Also, the National Institute on Aging and Alzheimer’s Association (NIA-AA) considers brain atrophy as a biomarker in the biological cascade of Alzheimer’s Disease¹⁰. Automated brain volumetry and its comparison to healthy, age-specific reference cohorts has been promised to assist clinicians in the diagnosis of dementia by highlighting the severity and distribution of brain atrophy in patients^{11,12,13,14,15,16}. Such quantitative reports (Q-reports)⁷ are offered by various proprietary tools. Additionally, longitudinal volumetric approaches have shown to be able to differentiate patients with MCI from normal aging¹⁷, which can, therefore, potentially be used as a biomarker for diagnosing the diseases or its progression. Still, the current Canadian consensus conference on the diagnosis and treatment of dementia, advises against the routine clinical use of automated quantification software, until larger studies demonstrate their added diagnostic value¹⁸.

Therefore, the aim of this study was to investigate whether volumetric measurements generated by a well-established automated tool (AI-Rad Companion Brain MR version VA40 [AIRC]^19,20, Siemens Healthineers, Forchheim, Germany) reflect brain atrophy in the same way as it is perceived during conventional visual assessment by expert neuroradiologists. To evaluate the robustness of longitudinal volumetric assessment, the qualitative and quantitative normalized volumetric differences (deltas) between baseline imaging and follow-up imaging were assessed in a random subset of twenty patients from a dementia outpatient clinic of a tertiary university provider.

2. Materials and Methods

3. Results

3.4. Group comparisons between expert visual assessment (EVA) scores and automated brain volumetric analyses (AVA) delta

Results of EVA scoring and delta of AVA in percentage are displayed in Table 2. There was a large spread of deltas between the two MRI surveys for each of the seven regions that are relevant for the diagnosis of dementia.

View this table:

• View inline
• View popup
• Download powerpoint

Table 2. Summary of the investigated brain regions comparing expert visual assessment (EVA) scores and AI-based brain volumetric analyses (AVA).

For each of the seven key anatomical regions, we calculated a global non-parametric test score to correlate EVA-scores with normalized region-specific AVA deltas. The link between EVA and AVA for the frontal lobe (p_KW=0.63, Figure 1a) could not be statistically verified. We found weak significance (p_KW=0.067, Figure 1b) for the parietal lobe. The right temporal lobe (p_KW=0.11, Figure 1c) did not show significant association, while there were strong associations for the for the left temporal lobe (p_KW=0.0092, Figure 1d) and the right hippocampus (p_KW=0.035, Figure 1e). There was no association for the left hippocampus (p_KW=0.12, Figure 1f). Lastly, we found strong association for the ventricle system (p_KW=0.0091, Figure 1g) between EVA and AVA. The delta differences between ventricle size (p=0.0002) for EVA score 0 and 2 stayed significant after Bonferroni and/or Holm-correction (p*=0.0042).

• Download figure
• Open in new tab

Figure 1.

Combination figure of expert visual assessment (EVA) scoring of atrophy between baseline and follow-up vs. automated volumetric analysis (AVA). The x-axes depict EVA grouped as score 0: no perceptible atrophy; score 1: probable progressive atrophy; score 2: certain progressive atrophy for the (a) frontal-, (b) parietal-, (c) right temporal-, (d) left temporal lobe, the (e) right and (f) left hippocampus while for the (g) ventricles score 0: represented no change, score 1: probable enlargement, score 2: certain enlargement, score -1: probable narrowing, and score -2: certain narrowing. Y-axes depict normalized difference (delta [%]) of AVA measurements between baseline and follow-up imaging. Kruskal-Wallis test (KW) revealed significant global associations between EVA and AVA-deltas for the left temporal lobe (d, p_KW =0.0092), the right hippocampus (e, p_KW =0.035), and for the ventricle volume (g, p_KW =0.0091). Pairwise comparisons (dashes) between EVA scoring levels 0 and 1 were significant for left temporal lobe (d, p=0.039) and right hippocampus (e, p=0.023). Pairwise comparisons between EVA scoring levels 0 and 2 were significant for the parietal lobe (b, p=0.013), left temporal lobe (d, p=0.0059) and ventricle volume (g, p=0.0002). Bold indicates significance at p* after Bonferroni-Holm-corrections. Pairwise comparisons were not performed for the (a) frontal lobe, (b) parietal lobe, the (c) right temporal lobe and the (f) left hippocampus since the KW test did not reach significance.

As sensitivity analyses, when combining EVA suspected and certain progression (scores 1 and 2) vs. no atrophy (score 0) progression, post hoc tests revealed a link between the left hippocampus (p_WSRT=0.039), right hippocampus (p_WSRT=0.0072), left temporal lobe (p_WSRT=0.0012) and ventricle volume (p_WSRT=0.0076). The parietal lobes (p_WSRT=0.064) showed a weak trend towards a potential association while the right temporal- (p_WSRT=0.13) and the frontal lobes (p_WSRT=0.43) missed significance.

4. Discussion

In this real-life explorative study, we investigated if expert visual assessment of longitudinal brain atrophy would be reflected by quantitative volumetric measurements produced by the AIRC-tool, an FDA-cleared, commercially available volumetric software application. Our study described three key findings regarding EVA-based atrophy progression assessment vs. longitudinal AVA: (1) we found a strong association between atrophy assessment by human experts and the normalized deltas of automated volumetric measurements, particularly for the right hippocampus, left temporal lobe and the ventricle system; (2) automated volumetric deltas between imaging timepoints showed high variance and were often not reflected by EVA-scores; although (3) EVAs by human experts showed strong inter-rater agreement. Hence, our findings underline the importance of carefully evaluating results of volumetric automated tools, if used for longitudinal assessment with varying acquisition protocols.

The significant association between EVA and AVA of the right hippocampus might be attributed to the highly standardized visual assessment procedure of the hippocampi, which showed high inter- and intra-rater reliability in a meta-analysis³⁰. Our results are also in line with results by Velickaite et al., where they showed significant association between the conventional brain atrophy score for medial temporal atrophy (MTA) and volumetric assessment in a cohort of 201 cases with well-preserved cognitive function at the age of 75 and 80 years³¹. Rau et al. also emphasized the robustness and clinical reliability of the MTA score³². Similarly, we found that besides the hippocampus, the anatomically closely associated left temporal lobe was significantly linked between human readers and the automated tool. Also, ventricle volume showed a strong association between EVA and AVA. This is relevant for patients with normal pressure hydrocephalus (NPH), a condition characterized by the accumulation of excess cerebrospinal fluid, leading to ventricular enlargement. NPH is one of the few treatable forms of dementia. Furthermore, Rau et al. showed that a machine learning algorithm could reliably detect NPH-patterns in 3D T1 MPRAGE images³³.

Visual assessment was found to be less sensitive than volumetric analysis using open-source or commercial tools^31,34. We found a weak correlation between EVA and AVA for the parietal and no association for the frontal lobe. This might be because, in our cohort, the median age was lower (66 years) than in other studies³¹. Also, median imaging follow-up time was 4 years in the whole cohort, and slightly longer in the MCI-compared to the neuropsychiatric disease group. This might have affected expert performance as longer follow-up periods increase the likelihood of detecting atrophy progression by EVA, as previously described for MTA scoring in the hippocampus³⁴.

The lack of association between EVA and AVA in certain regions might be attributed to inherent shortcomings of visual assessment, and might be particularly insensitive for the frontal and parietal lobes³¹. In contrast, inter-rater agreement was highest for the frontal lobe in our cohort.

When comparing AVA-results generated by the AIRC tool to measurements from the open-source FreeSurfer tool in 45 patients with de novo symptoms of memory decline, Rahmani et al. found excellent-to-good intraclass correlation consistency between the two tools in measured absolute volumes. They concluded that the AIRC-tool reliably detects atrophy in cortical and subcortical regions relevant for diagnosing dementia³⁵.

The metric used in this study, the percentage delta of the absolute volumes by the AVA, may produce higher numeric values for smaller brain regions due to its inverse proportionality to the baseline volume. Whitwell et al. suggested normalizing AVA measurements to total intracranial volume (TIV) for both cross-sectional and longitudinal studies to account for inter- and intraindividual image differences³⁶. They found that TIV normalization reduced inter-image differences caused by voxel-scaling variations from 1.3% to 0.5% (p=0.002). Although this reduction was statistically significant, the degree of variation was quite small. Thus, it should not have introduced substantial error into our results. Moreover, the right hippocampus, despite being a small structure, emerged as one of the most robust anatomical region in terms of consistency between EVA and AVA. Furthermore, its variance for each EVA scoring level (Table 2. Summary of the investigated brain regions comparing expert visual assessment (EVA) scores and AI-based brain volumetric analyses (AVA).) stayed comparable to large lobar structures and was smaller than for the entire ventricle system.

Zaki et al. compared two different AI algorithms for normative brain volumetry in 60 MCI patients and 20 controls. They found that different algorithms can have distinct effects that impact clinical interpretation, when used in isolation. Furthermore, they concluded that these AI-tools are not interchangeable during follow-up and need internal evaluation before adoption³⁷. Recently, many of these commercial AI-tools despite being CE/FDA approved, lacked proper clinical validation, especially for in-use evaluation⁷. Pemberton et al. showed that the use of Q-reports alongside visual assessment improved sensitivity, accuracy, and inter-rater agreement for the detection of volume loss¹³. They found that Q-reports were most useful for consultant-level radiologists, implying that more experience is required to benefit from information provided by quantitative analyses. Similarly, our findings showed that longitudinal AVA-based assessment may exhibit unexpected variability and should not be evaluated in isolation, in particular, when different acquisition protocols are being used. The results must be carefully inspected regarding technical limitations on a case-by-case basis. In a subset of cases in which EVA did not show atrophy progression in the frontal lobe and hippocampi, we found substantial variations with improbable volume gains and losses. This variation was >±10% for the frontal lobe and >±20% for the left hippocampus during a median follow-up of four years. In our experience with AIRC, the frontal and less often the parietal- and the temporal lobes were prone to segmentation instability of different severity. Some of these inconsistencies might be caused by skull-stripping, for which various approaches have been described and implemented^38,39. Consequently, parts of the brain may not be recognized as parenchyma and are not incorporated into the volumetric measurements (i.e., “minus-variant”, Supplementary Figure S1). In other cases, parts of the skull, dura, falx cerebri and venous sinuses were additionally incorporated leading to a volumetric surplus (i.e., “plus-variant”, Supplementary Figure S1). To note, however, that the most severe cases were almost exclusively “minus”-variants, particularly, in the frontal lobe. Therefore, to rule out segmentation errors, quantitative analytics results should be inspected thoroughly and discarded if “minus” or “plus”-variants are observed. This leads to the question what margin of error can be considered acceptable, when published annual atrophy rates based on normative populational data are around and under 1%. In their meta-analysis, Fraser et al. showed a mean hippocampal atrophy rate as low as 0.85% per year in a sample of n=3422⁴⁰. Sluimer et al. published annualized whole brain atrophy rates of -0.5% for healthy controls and -1.2% for patients with MCI⁴¹. If employed in longitudinal assessment the volumetric results from AIRC grossly surpass anticipated atrophy rates for the follow-up duration, or even suggest a volumetric gain, caution is advised during interpretation.

This high variance might also be explained by methodological and technical factors that can impact automated voxel-based morphometry. These factors have been previously described in detail including field strength, image resolution, acquisition sequence and image quality⁴². A study by Haller et al. showed that basic sequence parameters systematically bias volume estimation³⁶. Huppertz et al. investigated intra- and inter-scanner variability of brain volumetry multiple scans of a young, healthy volunteer in six different scanners⁴³. Reproducibility was best when the same sequence-protocol was performed on the same MRI-scanner. In our study, the use of different sequence-preferences and different MRI-machines may have impacted the volumetric results of the AIRC-tool. Although all three scanners were manufactured by the same vendor, even scanning on the same MRI-machine for baseline and follow-up did not fully rule out implausible AVA results. Scanning twice on Prisma produced more plausible results than scanning twice on Biograph. Notably, in patients with short follow-up time, in whom no severe atrophy progression was expected and accordingly none was detected during EVA, we found considerable variance of intraindividual regional volumetric changes. This might be counteracted by establishing reference atrophy ranges for image pairs using regularization techniques and Bayesian analytics, particularly for short term follow-ups of < 1 year⁴⁴.

Hydration status is also known to influence brain volumetry results^{45, 46, 47}, which might need further investigation in the context of AIRC robustness.

The present study has certain limitations, as it is a retrospective, explorative study of small size. Thus, it might be underpowered to confirm associations for all regions that are relevant for the diagnosis of dementia. However, it represents a random cohort of the daily practice. Similarly, all the imaging studies were performed on MR-scanners from the same vendor (Siemens Healthineers), but inevitably additional variance was introduced since follow-up images were acquired on various MRI scanners, as expected in the daily clinical praxis. Additionally, scan settings were not ADNI-conform, which potentially further limits the accuracy of the investigated proprietary AIRC tool. To note, however, that non-ADNI-conform settings represent the reality of many imaging centers and praxes. Lastly, the metric of percentage delta of AVA inherently yields higher numeric differences in smaller brain regions. However, the AVA-delta variances of the hippocampus were comparable to or even smaller than those of lobar structures or the entire ventricle system, regardless of the EVA-scoring level.

In conclusion, we found substantial and robust associations between human expert visual assessment and atrophy progression measured by the AIRC-tool for the right hippocampus, left temporal lobe and the ventricle system. Despite EVAs showing strong inter-rater agreement between human readers, like for the frontal lobe, normalized automated volumetric differences (deltas) between imaging timepoints often did not align with EVA-scores and exhibited substantial variance, even implying volume gain. Hence, our findings highlight the importance of carefully evaluating volumetric results of AIRC, when used for longitudinal assessment, especially when volumetric deltas exceed expectable atrophy rates for the duration of the follow-up. Caution is advised and results should be critically reviewed, especially when acquisition protocols vary across scans.

5. Data availability statement

The dataset used in this study is available from the corresponding author upon reasonable request.

Authors’ contributions

M.G. and M.E.M conceptualized the study. M.G. collected imaging data. L.H., L.F. provided clinical data. M.G. and H.W. performed visual image analysis. M.G., C.GC., H.W., M.E.M performed volumetric data extraction. C.W., M.G., M.E.M. performed statistical analyses. M.G., C.W., N.S., H.W., M.E.M. analyzed and interpreted the data. M.G. created figures. L.H., L.F., A.F., J.F., C.G. advised clinical aspects of the study. M.G., H.W., and M.E.M wrote the manuscript. H.W. and M.E.M. supervised the study. All authors critically reviewed the manuscript and approved the final version.

Funding

M.E.M. and N.S. reports funding from the German Ministry for Education and Research (BMBF) within the framework of the Medical Informatics Initiative (MII) MIRACUM Consortium (Medical Informatics for Holistic Disease Models in Personalized and Preventive Medicine, MIDorAI; 01ZZ2020). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflicts of interest

M.E.M. and H.W. report consultancy for Siemens Healthineers GmbH. M.E.M. reports unrelated consultancy for EppData GmbH. The remaining authors have no conflicts of interest to declare.

Figure legends

• Download figure
• Open in new tab