Detection and segmentation of hyperdense middle cerebral artery sign on non-contrast CT using artificial intelligence

Study population

For training and internal validation datasets, we retrospectively collected data of patients with ischemic stroke (admitted between May-2011 and December-2022) across six comprehensive stroke centers, using the following criteria: (1) age≥18 years; (2) admission within 7 days of symptom onset; and (3) concurrent NCCT and CTA acquisition within 10 min interval between exams.

For the external validation dataset, patients with ischemic stroke (admitted between July-2010 and November-2022) were retrospectively collected from one comprehensive stroke center, adhering to the aforementioned criteria.

To clinically validate our algorithm, we retrospectively collected a consecutive series of 941 patients with ischemic stroke or transient ischemic attack who were not overlapped with training and internal validation, and external validation datasets between December-2020 and April-2023 from six comprehensive stroke centers using the following criteria: (1) age≥18 years; (2) admission within 24 hours of symptom onset; and (3) concurrent NCCT and CTA acquisition within 10 min interval between exams.

In all cases, patients underwent both NCCT and CTA concurrently to evaluate LVO at the discretion of attending physicians. Patients were excluded based on the following criteria: (1) presence of brain tumors, intracranial hemorrhages, or surgical implants such as ventriculoperitoneal shunts or extraventricular drainage catheters, (2) NCCT scan exhibiting residual contrast agents, (3) NCCT scan with poor image quality, and (4) bilateral LVO (Figure 1). The institutional review boards of Seoul National University Bundang hospital approved the study and waived informed consent due to the anonymous and retrospective study design. The data that support the findings of this study are available from the corresponding author upon request. The study design adheres to the STARD guidelines (Standards for Reporting of Diagnostic Accuracy Studies).¹⁰

• Download figure
• Open in new tab

Figure 1.

Study flow chart. NCCT=noncontrast CT; CTA=CT angiography; LVO=large vessel occlusion; HAS=hyperdense artery sign

Clinical data collection

Using standardized protocol, demographic data, medication histories and details on vascular risk factors were collected.^11,12 Stroke subtypes were determined by consensus among neurologists at each participating center using a validated magnetic resonance imaging (MRI)-based algorithm. Admission National Institutes of Health Stroke Scale (NIHSS) scores, pre-stroke modified Rankin Scale (mRS) scores, and 3-month mRS scores were evaluated by certified (http://www.stroke-edu.or.kr/) physicians in each hospital.

Hyperdense artery sign annotations

Identification of HAS on thick-section NCCT scan alone is prone to misdiagnosis due to calcification, partial volume averaging, high hematocrit value, or cases with spontaneous recanalization.¹³ Thus, only patients who are confirmed to have LVO based on concurrent CTA were subjected to HAS annotation. Scans were adjusted to a window width of 80 HU and a center level of 35-40 HU. The presence of HAS was determined visually, meaning the decision was based solely on visual assessment without using any Hounsfield unit (HU) threshold as previously described.⁴ Manual segmentation of clots on NCCT was performed independently by two experts (with 8 and 20 years of experience, respectively) without referencing CTA. This approach ensured diversity in the training data by incorporating the perspectives of different experts.

Identification of LVO

In this study, we defined LVO as a complete contrast filling defect in intracranial carotid artery or middle cerebral artery M1/M2 segment. The presence, side, and location of LVO was determined referring to CTA which was concurrently performed with NCCT by an experienced neurologist. Subsequently, these diagnoses were compared with the stroke registry data, which have been independently verified by vascular neurologists at each comprehensive stroke center. A consensus was made if discordance in diagnosis was found. All data were analyzed in a central image laboratory.

Data preparation

In the training and validation dataset, patients from six stroke centers were randomly split in a 9:1 ratio into training and internal validation datasets. HAS segmentation masks were independently obtained by two experts. To simulate the inherent variability in HAS identification on NCCT scans, a new set of labels was created by randomly mixing the annotations from the two experts (Figure 1). To be specific, given a dataset of NCCT scans with corresponding annotations from both expert 1 and expert 2, a new set of clot sign segmentation masks was generated by randomly selecting annotations from either expert 1 or 2 for each case. To comprehensively capture the variability and potential disagreement between the experts, five additional sets of HAS annotations were constructed using this random selection process. This resulted in a total of seven sets of annotations, including the two original annotations from both experts.

Deep Learning Segmentation Model Training and Evaluation

We trained a deep learning-based semantic segmentation model using a combination of Dice loss and binary cross-entropy loss functions. The model architecture was based on a 2D U-Net¹⁴ with modifications to the number of down-sampling stages, feature dimensions, and normalization methods. As input, three consecutive slices of NCCT scan were windowed (width:50, level:60), resized to 256×256 pixels, normalized, and stacked into a three-channel image. The model outputs pixel-wise probabilities of HAS.

To prevent overfitting and maximize data variability, we applied various data augmentation techniques, including rotation, horizontal flipping, translation, brightness adjustment, zooming, noise addition, and random erasing. Detailed information on the model structure and training procedures is provided in the Supplemental Material (Figure S1).

We developed three different deep learning models in this study. Model 1 and Model 2 were trained solely using the data annotated by Expert 1 and Expert 2, respectively. To develop an ensemble model trained on data annotated by different experts, we trained the model on each set of NCCT scans and their corresponding annotations, resulting in seven distinct models. Model selection was performed using the internal validation dataset based on the Dice similarity coefficient (DSC). An ensemble method was then employed by averaging all pixel-wise predictions from the seven models. The performance of the ensemble model was compared to the models trained on single datasets annotated by each expert, referred to as Model 1 and Model 2, respectively. The metrics used to evaluate the models are as follows (detailed definitions are provided in the Table S1):

• Volume-based metrics: Spearman’s correlation for volume and absolute volume difference (AVD) [mm³]

• Overlap metrics: DSC

• Diagnostic metrics: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and predicted pixel-based area under the receiver operating characteristic curve (AUROC)

Additionally, to evaluate the robustness of the model’s performance and the inherent variability between experts, we assessed the consistency of model predictions on the internal and external validation datasets with manual segmentations by the two experts. For this analysis, only patients confirmed to have LVO were included, as no annotations were performed on patients without LVO. In cases where the prediction of HAS was correct but the location was incorrect (DSC=0), the case was designated as a false negative.

To validate the feasibility of our algorithm for use in a clinical setting, we conducted an analysis of its processing time. We randomly selected 100 NCCT scans, each with a slice thickness of 5mm, from the clinical validation dataset. The total processing time measured encompasses stages of reading DICOM files, pre-processing, and post-processing. The test was conducted on limited hardware resources (VRAM of 2GB and 8 CPU cores).

Clinical validation dataset assessment

In the clinical validation dataset, the presence of LVO was determined by an expert. NCCT scans were independently reviewed by two experts, and the presence of HAS was defined when both experts agreed. The length of the clot predicted by the algorithm was defined as the maximal diameter of the clot in the 2D plane. Infarct volume on DWI was automatically calculated using validated software (JLK-DWI, JLK Inc., Seoul, Korea)^15,16 and the segmented infarcts were meticulously reviewed by an expert to ensure correct segmentation. Information on the endovascular procedure including modified Thrombolysis in cerebral infarction (mTICI) scale after endovascular treatment (EVT), and 3-month modified Rankin scale (mRS) were retrieved from a prospective stroke registry.

Statistical Analysis

To compare baseline characteristics between the training, internal, and external validation datasets, we used ANOVA or the Kruskal-Wallis test for continuous variables, and the chi-square test or Fisher’s exact test for categorical variables, as appropriate. For comparisons of inter-expert agreement with model-expert agreements, we employed the paired t-test of the mean metric values. Confidence intervals for the evaluation metrics were calculated using bootstrapping with 1000 iterations. In this analysis, the number of predicted voxels was used as the criterion. Spearman’s correlation analysis was used to assess the clot volume correlation between manual and automated segmentations. In the clinical validation dataset, the frequency of HAS between stroke subtypes was compared using the chi-square test. Associations of the predicted clot length and volume with procedure time in patients undergoing endovascular procedures were analyzed using Pearson’s correlation analysis. Additionally, patients undergoing the procedure were stratified into two groups based on post-procedure mTICI scores (0, 1, and 2a versus 2b and 3), and length and volume were compared using the t-test. The association of predicted clot length or volume with infarct volumes on DWI was analyzed using Pearson’s correlation analysis. Furthermore, clot length and volume were compared by 3-month mRS scores using ANOVA. All statistical analyses were performed using STATA 16.0 (STATA Corp., Texas, USA), and a p<0.05 was considered statistically significant.

Baseline characteristics

Following the exclusion criteria, 45, 8, and 172 patients were excluded from training and validation, external validation, and clinical validation datasets (Figure 1), respectively, remaining 673, 365, and 774 patients. The mean±SD ages were 68.8±13.2, 67.6±13.4, and 68.8±13.6 years and frequencies of male were 55.0%, 59.5%, and 57.6%, respectively (Table 1). LVO was more frequent in external validation datasets than other datasets (59.4%, 65.2%, and 24.2% in training and validation dataset, external validation dataset, and clinical validation dataset, respectively; p<0.0001). Accordingly, HAS were more frequent in the external validation dataset regardless of which experts performed the labeling (p=0.038). CT vendors and imaging parameters were significantly different between datasets, indicating heterogeneity of datasets. Slice thickness of NCCT scans ranged from 3.0 to 5.0mm.

Performance of deep learning models detecting and segmenting hyperdense artery sign

In the internal validation dataset, the models showed sensitivity of 0.630~0.877, specificity of 0.702~0.835, and AUROC of 0.727~0.889. In the external validation dataset, the models showed sensitivity of 0.729~0.817, specificity of 0.595~0.826, and AUROC of 0.755~0.839. For both datasets, the ensemble model showed a slightly higher AUROC compared to the other two models (Table 2).

In the internal validation dataset annotated by expert 1, the ensemble model achieved a DSC of 0.611 (95% CI, 0.545−0.678), which was comparable to Model 1 and Model 2 (p>0.05). In the dataset annotated by expert 2, the ensemble model achieved a DSC of 0.686 (0.627−0.746), which was higher than Model 1 (p<0.001) and comparable to Model 2 (p>0.05). Similar results were observed in the external validation dataset, with the ensemble model achieving higher or comparable DSCs compared to Model 1 and Model 2. In the external validation dataset, DSCs between the ensemble model and each expert (0.622 [0.596−0.648] for expert 1 and 0.606 [0.579−0.633] for expert 2) were comparable to the inter-expert DSC (p > 0.05).

The volumetric correlation between the ensemble model’s outputs and expert annotations was higher than the inter-expert correlation (0.573), with model-expert correlations of 0.673 for expert 1 and 0.627 for expert 2 (Figure 2). The AVD between the ensemble model and each expert was lower than the AVD between the experts, with model-expert differences of 45.78 (±66.01) mm³ and 54.21 (±79.38) mm³ for experts 1 and 2, respectively. The mean processing time for segmenting HAS using the algorithm was 3.18 ± 0.12 seconds on NCCT scans with 4mm thickness (n=100).

• Download figure
• Open in new tab

Figure 2.

Volumetric analysis comparing clot volumes from manual annotation and automated segmentation algorithm. Dot plots for clot volumes by two experts (A), expert 1 and the algorithm (B), and expert 2 and the algorithm (C). Lines and shaded areas indicate regression lines and their 95% confidence intervals

Inter-expert agreement on hyperdense artery sign

In the external validation dataset, Cohen’s kappa and percent agreement for the presence of HAS between experts were 0.521 and 76.5%, respectively. The DSC between experts for HAS segmentation in patients marked as HAS-positive by both experts was 0.622 (95% CI, 0.594−0.650). For patients marked as HAS-positive by either or both experts, the DSC between experts was 0.312±0.314. Spearman’s correlation coefficient and AVD for manually segmented clot volumes between experts were 0.573 and 57.09(±78.99) mm³, respectively.

Review of false positive and false negative cases by deep learning algorithm

In the external validation dataset, the algorithm detected 71 false positive HAS. Among these, 24 showed symmetrically increased density in the bilateral MCAs, 24 displayed asymmetric density between the bilateral MCAs, 16 had calcified M1-MCA segments, 5 had relatively high attenuated MCAs due to adjacent chronic ischemic lesions, and 2 exhibited cortical high attenuated lesions (Figure 3). Additionally, the algorithm missed HAS in 27 patients. Among these, 11 had ambiguous density differences between the bilateral MCAs, 9 had HAS in the MCA-M2 or MCA-M3 segments, 3 had HAS in the calcified MCA, and 4 had short segment clot signs in the MCA.

• Download figure
• Open in new tab

Figure 3.

Representative false positive and false negative cases in the external validation dataset. (A) The artificial intelligence (AI) model detected calcification of the middle cerebral artery (MCA) (the first panel), high attenuation adjacent chronic lesion (the second panel), high attenuation area in the cortex (the third panel), and asymmetric density between the bilateral MCAs (the fourth panel) as hyperdense artery sign (HAS). (B) The algorithm missed HAS at the M2-MCA segment (the first panel), in the calcified MCA (the second panel), at short segment of the MCA (the third panel), and ambiguous density differences between the bilateral MCAs (the fourth panel).

Clinical validation of clot segmentation model

In the clinical validation dataset (n=774), 187 (24.2%) patients had LVO, and 69 patients (8.9%) had a HAS. The AUROC, sensitivity, specificity, PPV, and NPV of the algorithm were 0.846 (95% CI, 0.819–0.871), 0.768 (0.651–0.861), 0.885 (0.859–0.908), 0.396 (0.339–0.455), and 0.975 (0.962–0.984), respectively. The median length and volume (interquartile range) of automatically segmented clots were 8.2 mm (5.6−17.1 mm) and 116.8 mm³ (74.4−201.1 mm³), respectively. When patients (n=118) undergoing EVT were divided into two groups (mTICI 0, 1, or 2a vs. 2b or 3), both the length and volume of the clot tended to be lower in the latter group compared with the former group, although the differences were not statistically significant (p=0.57 and p=0.19, respectively; Figure S2). In patients receiving EVT, no association was found between algorithm-detected clot length or volume and the procedure time of EVT (Figure S3).

In patients with follow-up DWI (n=702), infarct volumes on DWI were higher in the HAS-positive group than in the HAS-negative group (median [IQR] 6.98 [0.43−75.01] mL vs. 1.16 [0.21−6.87] mL, p<0.001). Additionally, the predicted length and volume of the clot were significantly correlated with infarct volume (r=0.34 and 0.42, respectively; both p<0.001; Figure 4A and 4B). When we restricted the analysis to patients with LVO (n=171), similar associations were observed (r=0.31 and r=0.36, respectively; both p<0.001; Figure 4C and 4D). Regarding the mRS score at 3 months after stroke, both the predicted length and volume of the clot tended to increase as the mRS score increased in the entire population (Figure S4). In patients with LVO, we observed a trend of increased predicted length and volume of the clot, although the difference was not statistically significant.

• Download figure
• Open in new tab

Figure 4.

Correlation of length and volume of segmented clot and infarct volumes on follow-up diffusion-weighted image (DWI) in the clinical validation dataset. Correlation of clot length (A) and volume (B) with infarct volume on DWI in all patients. Correlation of clot length (C) and volume (D) with infarct volume on DWI in patients with large vessel occlusion. Lines and shaded areas indicate regression lines and their 95% confidence interval.