A multimodal neural signature of face processing in autism within the fusiform gyrus

Sample characterization

Participants were part of the EU-AIMS/AIMS-2-TRIALS LEAP cohort^58,59. They underwent comprehensive clinical, cognitive and MRI assessment at one of six collaborating sites. All autistic participants had an existing clinical diagnosis of autism which was confirmed using the combined information of gold-standard diagnostic instruments, the Autism Diagnostic Interview-Revised⁶² (ADI-R) and the Autism Diagnostic Observation Schedule⁶³ (ADOS). The study was approved by the respective research ethics committees at each site (IRAS, UK). Informed written consent was obtained from all participants, or—for minors or those unable to give informed consent—from a parent or legal guardian. For further details on diagnostic procedure, study design and exclusion criteria, see Supplemental Information (SI) and our earlier papers^58,59. The final sample has both complete imaging data across four different imaging modalities that were integrated (described below) and phenotypic information available. This consisted of 99 autistic individuals, and 105 NAI between 7 and 30 years. For details on demographic information, see Table 1.

Clinical and cognitive measures

We split available autism-associated measures into two sets of feature sets based on the construct they measure 1) social-communicative features comprising measures of difficulties with social communication and daily living skills (i.e., ADOS-social affect, ADI-communication, ADI-social, Vineland Adaptive Behavior Scale⁶⁴ with Communication, Daily Living, Socialization subscales), emotional face matching performance (i.e., Hariri faces task⁶⁵), and social sensitivity to complex emotions (i.e., Reading the Mind in the Eyes test⁶⁶ [RMET]) and 2) non-social features comprising restricted, repetitive behaviours (RRBs) (i.e., ADOS-RRB, ADI-RRB, the Repetitive Behavior Scale⁶⁷ [RBS-R]), systemizing (i.e., the Systemizing Quotient^68–70 [SQ]), shape matching performance (i.e., Hariri shapes task, as the control condition to the Hariri emotional faces task) and sensory processing atypicalities (i.e., Short Sensory Profile⁷¹ [SSP]) (see Supplement and Table S1). To tackle missing clinical data and to not further reduce sample size, we used imputed clinical data⁷², as in previous work with this dataset^56,73.

Region of interest: fusiform gyrus

All analyses were restricted to the right and left FFG based on the Harvard-Oxford atlas (HOA) (FMRIB, Oxford, UK) (i.e., anterior and posterior divisions of the temporal fusiform cortex, temporal occipital fusiform cortex and occipital fusiform gyrus). The size of the ROIs was adjusted to have 100% coverage across all individuals for each imaging modality (for details see SI).

Imaging modalities

For MRI and EEG data acquisition parameters and detailed preprocessing steps, see SI and Table S2 and S3.

Normative modelling

Normative modelling is an emerging statistical technique that allows parsing heterogeneity by charting variation in brain-behaviour mappings relative to a normative range and provides statistical inference at the level of the individual⁸⁰. The term ‘normative’ should not be seen as incompatible with the neurodiversity framework as it simply refers to statistical norms such as growth charts that vary by demographics such as age and gender. Variation from the norm is part of neurodiversity. Here, we trained normative models^60,80,81 using Bayesian Linear Regression (BLR)⁸² (https://pcntoolkit.readthedocs.io/en/latest) for each brain imaging modality within the right and left FFG ROI independently using age, sex and scanning site as covariates. A B-spline basis expansion of the covariate vector was used to model non-linear effects of age. Normative models were derived in an unbiased manner across the entire sample under 10-fold cross-validation^52,60,83. This Bayesian approach calculates the probability distribution over all functions that fit the data while specifying a prior over all possible values and relocating probabilities based on evidence (i.e., observed data). As such, it yields unbiased estimates of generalizability and inferences with increasing uncertainty with fewer data. To estimate voxel-wise/time-point-wise deviations for each modality in each individual, we derived normative probability maps (NPM) that quantify the deviation from the normative model summarized in Z-scores. These subject-specific Z-score images provide a statistical estimate of how much each individual’s recorded value differs from the predicted value at each voxel/time-point. The accuracy of the normative model was evaluated using the correlation between the recorded and the predicted voxel values (Rho), the mean standardized log-loss (MSLL), standardized mean squared error (SMSE), and the explained variance (EV) (Figure S1) as well as based on the forward models (Figure S3). Furthermore, we compared model performance when modelling age linearly (without a B-spline basis expansion; Figure S2). To assess whether autistic and NAI differed in their extreme deviations based on unimodal features, thresholded Z-scores (Z>|2.6|^30,52,84,85, corresponding to the 99.5th percentile) were compared between the two groups using a two-sample t-test (see SI). Code is available at https://github.com/amarquand/PCNtoolkit.

Linked Independent Component Analysis

In order to gain more comprehensive insights into cross-modal signatures of face processing, we merged the different individual-level deviations from all imaging modalities (GM volume, FFA-connectivity, T-maps contrasting the faces condition to the shapes condition, and the principal components of source reconstructed time series) using Linked Independent Component Analysis (LICA)^{47,49–52,56,86} (see SI). This is a Bayesian extension of the single modality ICA model which provides an automatic and simultaneous decomposition of the brain features into independent components (ICs) that characterize the inter-subject brain variability. These multiple decompositions share a mixing matrix (i.e., subject course) across individual feature factorizations that reflect the subject contributions to each IC. These subject loadings per IC were later used to investigate the multivariate relationship between the brain phenotypes and clinical measures (see canonical correlation analysis below). Further, each IC also provides a map of spatial or temporal variation per modality and a vector reflecting the relative contribution of each modality to the component. Here, LICA was used to merge the unthresholded Z-deviation maps derived from normative modeling across the four different imaging modalities within the right and left FFG ROIs. Each measure per hemisphere was treated separately (i.e., right structure, left structure, right rs-fMRI, left rs-fMRI, right task-fMRI, left task-fMRI, right EEG, left EEG) resulting in eight input maps (i.e., modalities). Hemispheres were modelled separately given known brain asymmetric differences in autism^30,44,87 and to study the hemispheric contributions and model the different noise characteristics individually. We estimated 50 independent components based on our sample size and following recommendations described in earlier papers^{50–52,56,86} (i.e., sample size ∼N / 4). To evaluate the robustness of our selected model order (N=50), we re-ran LICA using different dimensional factorizations of subject loadings (N=40 and N=60) and computed correlations among them. LICA code is available at https://github.com/allera/Llera_elife_2019_1/tree/master/matlab_flica_toolbox.

Group Differences

The subject loadings of all ICs were compared between autistic and NAI using a two-sample t-test. Multiple comparisons were corrected for using the False Discovery Rate (FDR)⁸⁸. ICs showing significant group differences were further characterized by plotting each contributing modality’s spatial map and temporal profile (Z-thresholded at the 95^th percentile). To further characterize the most implicated regions within the FFG per modality, we computed the overlap between supra-threshold voxels and a structural (i.e., the Harvard-Oxford atlas, which covers the entire FFG) and a functional (i.e., a probabilistic functional atlas of the occipito-temporal cortex¹⁸ which covers category-specific FFG patches) atlas (see SI).

Multimodal components

Next, given the current work’s focus on multimodal neural sources, we tested the hypothesis whether multimodal components performed superior to unimodal components in differentiating autistic individuals from NAI. For this, we calculated a multimodal index (MMI) per independent component to quantify the multimodal nature of modalities in each IC⁵⁰ (for details, see SI). The MMI ranges from 0 (equating to 100% unimodal contribution) to 1 (equating to equal contributions from all modalities). Multimodal components were defined as each single imaging modality (i.e., regardless of hemisphere) not having more than a 90% contribution to each component and an MMI below 0.1 (see Figure S4). Components below this threshold were regarded as unimodal.

Autism classification

Next, we implemented two support vector machine (SVM) classifiers with a linear kernel – one using unimodal and one using multimodal components as features to test for the added value of multimodal features. The SVM was trained and evaluated using 10-fold cross-validation and class-weighting was used to account for group size imbalance. The area under the receiver operating characteristic curve (AUC) was used as the performance metric to assess the classifier’s discrimination ability. To test for significant differences in AUC between multimodal and unimodal components, we generated a null distribution of AUC differences by shuffling the cross-validated scores 10.000 times and re-evaluating the classifier performance and computed the likelihood of observing the observed AUC difference under the null hypothesis. To test for robustness of results across different multimodal thresholds, we re-ran analyses across different thresholds resulting in slightly varying degrees of multimodality ranging between 85% to 99% of single modality contributions. Given that each threshold resulted in a different number of unimodal vs. multimodal components, we further checked whether results remained stable when forcing uni- and multimodal components to have the same number of features; for this we selected between the top one and 22 most uni- and multimodal ICs.

Clinical-cognitive associations

To test for the clinical relevance of multimodal ICs, we ran canonical correlation analyses (CCA)⁸⁹ modelling the multivariate relationship between multimodal ICs and cognitive, clinical features related to either social or non-social functioning (described in detail above) in autistic individuals only. CCA is a multivariate approach to simultaneously model two sets of linear projections (based on the brain-related independent components and the cognitive features) to maximize their correlation. The statistical significance of the CCA modes was assessed by a complete permutation inference algorithm proposed by Winkler et al.⁹⁰, where both brain and behaviour data were permuted separately across all participants with 10,000 iterations. In total, we ran two separated CCAs testing the multivariate relationship between the brain measures (multimodal ICs) and a) social-communicative features related to social functioning and face processing in autism or b) non-social features associated with autism. For further details, see the SI.

To visualize the spatial and temporal patterns of each imaging modality associated with each clinical cognitive measure, we computed the correlations between the original imaging data (i.e., the Z-deviation maps) and the canonical imaging variate (V) derived from the CCA⁹¹. Significance of correlation maps was assessed with 1000 permutations and significant clusters / timepoints were next visualized and further characterized in terms of their functional and anatomical characteristics by computing their overlap with the probabilistic functional atlas of human occipito-temporal visual cortex (VIS-atlas)¹⁸ of early visual and category-selective regions (see Figure 2h) and the HOA atlas (see Figure 2m). For further details see the SI.

• Download figure
• Open in new tab

Figure 1.

Overview of the methodological approach. Features for each modality were extracted from the right and the left fusiform gyrus. These were: a) grey matter volume based on VBM for structural MRI; b) T-maps contrasting the faces condition to the shapes condition reflecting sensitivity to emotional faces from the Hariri paradigm for task-fMRI; c) seed-based (i.e., fusiform face area) connectivity (SCA) for rs-fMRI; and d) the principal component of different source reconstructed time series for EEG. Next normative modelling was applied to each imaging modality using Bayesian Linear Regression. To model cross-subject individual-level variation, resulting Z-deviation maps per modality were statistically merged using linked independent component analysis resulting in measures of modality contributions and subject loadings. Next, we tested for group differences in ICs and group separability using either multi- or unimodal ICs and compared their performance. Finally, we computed multivariate associations (i.e., canonical correlation analysis) between subject loadings and clinical, cognitive measures related to either social-communicative or non-social features.

• Download figure
• Open in new tab

Figure 2.

Among all independent components, eleven were considered multimodal (Figure 2a), with a single modality contribution not more than 90%. Among these, IC44 showed a significant group difference where autistic individuals had higher contributions than non-autistic individuals (Figure 2b). Figure 2c shows spatial and temporal Z-maps thresholded at the 95^th percentile of the different neuroimaging modalities associated with IC44. Positive values (in yellow) depict positive loadings onto the IC and mean autistic individuals have higher deviations than NAI; whereas negative values (in blue) depict negative loadings onto the IC and mean autistic individuals have lower deviations than NAI. Suprathreshold timepoints are depicted in red. Figures 2d-g depict the spatial overlap of suprathreshold voxels with a probabilistic functional atlas of the occipito-temporal cortex (i.e., VIS-atlas¹⁸). Figure 2h depicts the VIS -atlas and the different early visual and category-selective subregions covering the FFG. Figures 2i-l show the spatial overlap of suprathreshold voxels with the structural Harvard-Oxford atlas and the four subregions of the fusiform gyrus (i.e., anterior and posterior divisions of the temporal fusiform cortex, temporal occipital fusiform cortex and occipital fusiform gyrus depicted in Figure 2m). Here, Figures 2d-e and 2i-j show the positive loadings (i.e., autism > NAI) and Figures 2f-g and 2k-l the negative loadings (i.e., autism < NAI), whereas Figures 2d/f and 2i/k depict the left hemisphere and Figures 2e/g and 2j/l the right hemisphere.

Finally, in order to assess robustness of CCA results, as previously, we set a range of multimodal thresholds between 85% to 99% and selected components with modality contributions exceeding this threshold as multimodal. We then re-ran the CCA for each threshold to assess stability of results across slightly varying degrees of multimodality.

Sample

The final sample of autistic (N=99) and NAI (N=105) did not differ significantly in sex ratio, age, measures of intellectual functioning, measures of structural image quality, number of EEG trials and head motion associated with task- and rs-fMRI (for details see Table 1).

Unimodal normative models

First, unimodal normative models were estimated. Their accuracy was evaluated using the correlation between the true and the predicted voxel values (Rho), the EV, SMSE and MSLL (Figure S1) and normative models per modality (Figure S3). Evaluation metrics were largely within recommended ranges⁹² and highly similar when modelling age linearly (Figure S2). When testing for group differences in unimodal features, there were no significant differences in extreme Z-deviations between autistic and NAI for any of the eight features (Table S4).

Linked independent component analysis

Next, the Z-deviations (features) were merged using LICA. Fifty independent components were derived across eight different brain feature maps per hemisphere (i.e., modalities) (Figure S5). Overall, across these, the right hemisphere (51.7%) and the left hemisphere (48.3%) did not contribute differentially (χ²=1.2, p=0.72). Single modality contributions were as follows: EEG R (35.0%) > EEG L (33.2%) > rs-fMRI R (11.2%) > rs-fMRI L (9.6%) > task-fMRI R (3.5%) > task-fMRI L (3.4%) > structure L (2.1%) > structure R (2.1%). Figure S6 shows the correlations between the 50-dimensional factorizations (y-axis) and alternative 40 (Figure S6a) and 60 (Figure S6b) dimensional factorizations. Most components were recovered with high accuracy independently of the order of the factorization. This is in line with previous reports⁵¹.

Group Differences

Next, we compared the subject loadings of all (uni- and multimodal) ICs to test for differences between autistic and NAI. Among these, one multimodal IC (#44) showed a significant group difference with autistic individuals having higher contributions compared to NAI (t=3.5, p_FDR=0.026) (Figure 2b). There were no significant group differences in the remaining ICs (see Table S5). The significant multimodal component was not differentially driven by the right (52.8%) or left hemisphere (47.2%) (χ²=0.4, p=0.51) and was associated with several functional modalities (rs- fMRI R [48.5%], rs-fMRI L [35.0%], EEG L [11.6%], task-fMRI R [3.3%]), and to a smaller extent with GM volume (structure R [1.0%], structure L [0.5%]). Figure 3a depicts the spatial and temporal patterns for each imaging modality within IC44. When characterizing these further in terms of their anatomical and functional overlaps with the HOA- and VIS atlases, in the left hemisphere, autistic individuals showed more functional deviations than expected in rs-fMRI connectivity primarily in retinotopic regions of occipital FFG, while to a smaller extent also in lower-order face-selective regions (IOG) (Figures 3b and 3f). In the right hemisphere, they showed linked increased deviations in rs-fMRI and structure primarily in higher-order face-(mFus, pFus) and bodies-selective (OTS) regions of temporal-occipital and occipital FFG (Figures 3c and 3g). On the other hand, regions in the left hemisphere where autistic individuals showed linked decreased deviations compared to NAI, localized to both higher-order face-selective (mFus, pFus) and retinotopic regions of posterior, temporal-occipital and occipital FFG (Figures 3d and 3h). In the right hemisphere, these were mostly in higher-order face face-selective regions (pFus) across rs-fMRI and task-fMRI and in retinotopic regions across structure in temporal-occipital FFG (Figures 3e and 3i). Furthermore, autistic individuals showed more left EEG source activation than expected around 195-203ms and 417-426ms, whereas less source activation at 444–449ms than expected. For further details see Table S6.

• Download figure
• Open in new tab

Figure 3.

The multivariate association (i.e., canonical correlation) was significant between the eleven multimodal ICs and the social-communicative features associated with autism. Figure 3a shows the loadings of each multimodal component contributing to the CCA mode, while Figure 3d shows the loadings of each social-communicative feature contributing to the CCA mode; stars show the significant loadings. Figure 3b shows the canonical correlation scatterplot color-coded by the highest contributing clinical feature (ADOS social). The x-axis depicts the projected behavioural CCA variate and the y-axis the multimodal ICs CCA variates. Figure 3c shows the modality contributions of the four ICs that contribute significantly to the CCA. Figure 3e depicts the spatial and temporal patterns of each imaging modality that are significantly correlated with the social-communicative features. These are based on the significant correlation values between the Z-deviations of each imaging modality and the canonical imaging variate derived from the CCA.

Multimodal components

For further analyses, we focused on multimodal components only by excluding those which were primarily driven by one imaging modality, resulting in eleven multimodal ICs (Figure 2a). Across these multimodal ICs, the right hemisphere (60.0%) contributed more than the left hemisphere (40.0%) (χ²=7.2, p=0.007). Single modality contributions across all multimodal ICs were as follows: EEG R (26.3%) > rs-fMRI L (19.7%) > EEG L (13.9%) > rs-fMRI R (12.1%) > task-fMRI L (9.8%) > task-fMRI R (9.6%) > structure R (4.4%) > structure L (4.2%).

Autism classification

Applying an SVM, multimodal components performed significantly better at discriminating autistic individuals from NAI (AUC unimodal=0.48, AUC multimodal=0.64, p<0.001). This result was confirmed across a range of different multimodality thresholds (see Figure S7a) and was not influenced by varying amounts of features between multimodal and unimodal ICs (Figure S7b).

Clinical, cognitive associations

The CCA analysis revealed a significant multivariate association between the multimodal ICs and social-communicative features (i.e., ADOS-social affect, ADI-social, ADI-communication, Vineland-Communication, Vineland-Daily Living, Vineland-Socialization, RMET, Hariri face matching condition) (r=0.65, p_FDR=0.008; Figure 3b). On the other hand, testing the relationship between the multimodal ICs and non-social features (i.e., ADOS-RRB, ADI-RRB, RBS, SSP, SQ, Hariri shape matching condition), did not yield any significant association (r=0.49, p_FDR=0.51; Figure S8) pointing to specificity with social-related features of multimodal ICs. These associations remained stable when varying the multimodality threshold (Figure S9). For the significant association, multimodal IC37 showed the largest contribution on the imaging side followed by IC38, IC44 and IC34 (Figure 3a and 3c), whereas ADOS social-affect, RMET and Hariri face matching scores showed the largest contribution on the behavioural side (Figure 3d). The ICs contributing most are depicted in Figures 2c and Figures S10-12. On average, the right (56.5%) and the left hemisphere (43.5%) did not contribute differentially to these four ICs (χ²=2.9, p=0.09) which were mostly driven by functional modalities (EEG, task- and rs-fMRI). Next, imaging patterns correlating with social-communication features were characterized in terms of their overlap with anatomical and functional overlaps with the HOA and VIS atlases (Figures 3e and S13). Especially in higher-order face-selective regions (mFus and pFus) of posterior and temporal-occipital FFG, there were both linked increased deviations in bilateral rs-fMRI and task-fMRI and linked decreased deviations in bilateral structure and right rs-fMRI connectivity. At the same time, particularly in retinotopic regions of occipital FFG there was more bilateral GM volume along with less right task-activation than expected. There were more deviations in right EEG source activation at around 290ms, while left EEG did not reach significance. These joint imaging patterns were associated with more social difficulties as assessed by the ADOS, ADI and Vineland and more errors on the RMET, while also with greater accuracy on the Hariri faces task. For more details, see Tables S6-10.

Group differences

Among all components, one multimodal component (i.e., IC44) showed a significant difference in subject loadings between autistic and NAI. Overall, the right and left hemisphere did not show differential contributions within this IC, and it was associated with all modalities fed into the analysis, with the functional modalities, especially rs-fMRI and EEG, contributing most (see Figure 2). Particularly, the overlap with the VIS-atlas highlighted that face-selective and retinotopic regions of the FFG were most different between autistic and NAI, again showing a differential pattern by modality and hemisphere. More specifically, in the right hemisphere, higher-order face-selective regions exhibited less task activation and FFA-connectivity than expected, primarily in occipital FFG areas (Figures 2g and 2l). At the same time autistic individuals showed increased deviations in FFA-connectivity primarily in temporal-occipital FFG along with increased GM volume deviations in higher-order face-selective FFG regions (Figures 2e and 2j). This strong right-hemisphere involvement of regions associated with FFA across several modalities is in line with reports of increased FFA volume³⁸ and decreased FFA task-activation^93,94 and FFA-connectivity^34,95 in autism. Similarly, temporally, autistic individuals also showed more positive left deviations around 195ms which can be indicative of the consistently reported finding of a slower N170 in autistic individuals³⁶. This has specifically also been shown and extensively characterised in the same sample³⁷. Together these patterns converge to point towards autism-associated differences in face-selective areas of the FFG, both at the structural, functional, and temporal levels. Although these results align with earlier unimodal discoveries, previously it was uncertain whether disparate signals would be separate or coalesce to a joint multimodal expression. In this context, we provide evidence supporting the interconnected nature of distinct signals within a single, unified framework.

On the other hand, in the left hemisphere, IC44-related increased deviations in EEG source activation at around 420ms may indicate reductions in the face-N400 which has been associated with familiar face recognition and semantic information^96,97. While in NAI face processing becomes the most highly developed visual skill, in autistic individuals faces may convey greater novelty and thus decreased familiarity. Furthermore, in the left hemisphere of IC44, occipital, retinotopic areas of the FFG were most implicated as shown by increased functional connectivity deviations between the FFA and retinotopic and lower-order face-selective areas of the FFG in autistic individuals (Figures 2d and 2i). This was echoed by less GM volume than expected in left hemisphere retinotopic areas of FFG in autistic individuals (Figures 2f and 2k). Retinotopic, early visual areas act as the first stage in a hierarchical network of face processing in which lower-level feature-based components are processed before more complex features in higher-order face-selective regions (e.g., pFus, mFus)⁹⁸. Neural deviations in early visual areas as seen here are in line with reports of autistic individuals showing differences in sensory processing at early perceptual stages and have been described at the cognitive level as weak central coherence⁹⁹. Accordingly, studies show that autistic individuals exhibit a different strategy in processing facial and visual stimuli with a stronger focus on featural, local aspects at the expense of holistic, global information¹⁰⁰. Similarly, fMRI studies converge to show greater feature-based perceptual strategies in autistic individuals who primarily tend to recruit object-related regions (such as inferior temporal gyrus¹⁰¹ or occipital cortex³²) when viewing facial stimuli. Taken together, this suggests that differences we discovered in the left-hemisphere point primarily to low-level, bottom-up processing differences, whereas in the right-hemisphere they may indicate higher-level atypicalities in the FFA, with a differential involvement across the different structural and functional modalities.

Clinical, cognitive associations

Multimodal ICs showed a significant multivariate association with a set of clinical and cognitive features associated with social functioning in autism (see Figure 3).

IC44 which also showed a significant group difference was among these significantly contributing ICs to this associations. Components mostly driven by functional modalities (i.e., EEG, rsfMRI and task-fMRI) loaded significantly onto the CCA. Right EEG source activation deviations were at around 280-300ms, potentially indicative of the N250r generated in the FFG¹⁰² and associated with repetition of familiar facial stimuli¹⁰³. The amplitude of the N250r has been shown to decrease with increasing working memory (WM) load¹⁰⁴. This would translate into increased deviations as seen in autistic individuals here and may imply differences in degrees of WM resources allocated to the processing of facial stimuli which in turn have a larger novelty character in autistic individuals requiring more attentional effort. With regards to the other modalities, increased deviations particularly in higher-order face-selective regions across brain function (task-fMRI and rs-fMRI) while also in lower-order early visual regions across brain structure were associated with more autistic features, such as more social difficulties as assessed by ADOS, and lower social sensitivity as assessed by the RMET. Previous unimodal studies for example showed that the delayed latency of the N170 predicts change in social adaptive behaviour in autistic individuals³⁷ (i.e., EEG), autistic individuals with low performance on facial emotion recognition have reduced bilateral FFG activation (i.e., task-fMRI)⁵ and atypical FFA-connectivity is associated with increased social symptom severity in autism (i.e., rs-fMRI)³⁴. Here, we extend unimodal results to a multivariate association across a range of social-communicative features that are related to cross-modal signatures within the FFG. Previously, it was uncertain whether these separate neural signals contribute orthogonally or jointly to social-communicative features in autism. Here, we provide evidence for an interrelated biological basis of core social functioning in autism and that appropriately modelled shared variance across different modalities increases sensitivity to clinical-cognitive features associated with autism⁴⁷. Remarkably, at the same time, there was no association with a set of non-social features, such as repetitive behaviours or sensory processing, pointing to specificity of these multimodal ICs with regards to social functioning.

Summary and implications

Taken together, the multimodal neural signature within the FFG in autism presents with differential effects across hemispheres, modalities, and topographic organization.

Specifically, the picture emerges that (i) the functional modalities contribute more than the structural modalities and (ii) retinotopic, posterior-occipital regions are more implicated in the left hemisphere and higher-order regions more implicated in the right hemisphere within the FFG when it comes to group differences; but they do not contribute differentially with regards to social functioning. (i) Concurrent neural activity and functional co-expressions (task-, rs-fMRI, EEG) were strongly tied to social features observed in autistic individuals at present (such as current performance and ADOS assessment). On the other hand, more stable structural aspects of the brain established over time and historical symptoms reported through the ADI and Vineland – which provide insights into past behaviors – had a comparatively smaller impact on the observed association. These results highlight the dynamic nature of the relationship between neural activity and social functioning in autism and underscore the importance of considering the temporal dimension when investigating the neural correlates of social functioning in autism. Putative future neuroscientifically informed interventions targeting social features may thus benefit from a focus on concurrent neural functioning. (ii) Topographically, the FFG is known to exhibit an anterior to posterior gradient with more posterior regions related to lower-order, early visual processing, and more anterior regions related to higher-order processing¹⁰⁵. Here, we see the involvement of both retinotopic and higher-order cognitive, particularly face-sensitive patches pointing to differences in both bottom-up perceptual processes and top-down cognitive information processing in face processing in autism which can amount to a difference in the face processing strategy employed (e.g., more feature-based). These different processing levels are not differentially implicated across hemispheres in the processing of social information in autism suggesting that the distinctive face processing strategy in autism transcends right hemisphere dominance of face processing. On the other hand, hemispheric differences are more apparent in the group-differential IC. Teasing apart hemispheric contributions is particularly important in the light of reports of atypical patterns of brain asymmetry in autistic individuals^30,44,106. More extreme deviations from a normative model have for example been reported in right temporal-occipital fusiform cortex asymmetry in autistic females³⁰, along with more left-lateralized volume in posterior temporal FFG in autistic individuals^40,44. Subsequent research should delve further into these more nuanced insights revealed by cross-modal analyses pointing to left-lateralized low-level and right-lateralized high-level differences between autistic and NAI.

Strengths and limitations

Integrating data from different modalities has the advantage of being biologically more informative and comprehensive in characterizing a complex, heterogenous condition like autism. Accordingly, when comparing unimodal deviations in each imaging modality, as well as comparing predominantly unimodal ICs between autistic and NAI, there were no significant group differences, despite employing a more sensitive individual-level measures derived from normative modelling. Also, multimodal features significantly outperformed unimodal features in differentiating autistic form NAI. These results together confirm our hypothesis and previous reports^47,48 that appropriately modelling cross-modal variance increases sensitivity to detecting subtle effects that may otherwise be missed. Thus, integrating different structural and functional brain measures is the most promising and powerful method to achieve significant advances in our understanding of system-level atypicalities in autism and provides the basis for elucidating mechanisms through which interventions can most efficiently improve clinically relevant functioning⁴⁷. Furthermore, we combine different innovative methods. LICA is particularly powerful when modelling modalities that are different in their numbers of features, spatial correlations, intensity distributions and units. This is, because LICA optimally weighs the contributions of each modality by the correction for the number of effective degrees of freedom and the use of automatic relevance determination priors on components^8,47,49. Also, by combining normative modelling with LICA, we employ a previously validated approach that has been shown to increase sensitivity in detecting cross-modal effects in clinical populations⁵². Future studies should compare these methods with alternative emerging approaches that aim to combine normative modelling with multimodal integration in a single step, such as the use of variational autoencoders for example¹⁰⁷.

At the same time, it needs to be pointed out that face processing involves an extended neural network across the whole brain including other structures such as the amygdala, superior temporal sulcus and occipital and frontal cortex^{32,52,108,109}. It may thus seem too simplistic to reduce face processing to a single brain region. Still, the FFG has been claimed the core node of a distributed face processing network, as also substantiated by FFG lesion studies^27,110, and its fine-grained functional heterogeneity warrants careful examination in isolation. Also, implementing cross-modal analyses presents with additional challenges, such as obtaining sufficiently large sample sizes with all participants having available data across all imaging modalities. Here, from a sample of over 600 individuals in the EU-AIMS LEAP dataset, we were able to conduct analysis in just over 200 individuals who had available imaging data across the four different modalities. Whole brain analyses based on multivariate techniques will ultimately require larger sample sizes.