Psychometric Properties of the NIH Toolbox Cognition and Emotion Batteries Among Children and Adolescents with Congenital Heart Defects

NIHTB Cognitive Battery

Participants completed subtests of NIHTB-CB, generating composite scores for Crystallized Cognition, Fluid Cognition, and Total Cognition. The subtests for Crystallized Cognition include the Oral Reading Recognition Test (ORRT) and the Picture Vocabulary Test (PVT). On the ORRT, a test of letter identification and word reading, participants were shown letters (for younger children) or words (for older participants) and asked to read them aloud (Gershon et al., 2013). On the PVT, which assesses receptive vocabulary, participants selected images matching descriptive audio cues. The test uses computer adaptive testing methods (Gershon et al., 2013).

The subtests for Fluid Cognition include the List Sorting Working Memory Test (LSWMT), the Pattern Comparison Processing Speed Test (PCPST), the Flanker Inhibitory Control and Attention Test (FIC+AT), the Dimensional Change Card Sort Test – Executive Function (DCCST), and the Picture Sequence Memory Test (PSMT). On the LSWMT, a test of working memory, participants ages 7 and older were presented with a series of items presented visually and auditorily and were then asked to repeat the presented items, ordering them based on particular criteria (e.g., item size). The test requires participants to hold a set amount of information in mind, mentally organize the list of items based on a given criteria, and then verbally recall the information (Tulsky et al., 2013). On the PCPST, a test of processing speed, participants were presented with two images on the screen and had to quickly indicate if the two images matched (smiley and frowny face buttons for those ages 6 and younger, yes and no buttons for those ages 7 and older) (Carlozzi et al., 2013). On the FIC+AT, participants were instructed to focus on a central image (fish for those ages 7 and younger; arrows for those ages 8 and older) flanked by additional stimuli, which may or may not point in the same direction. Participants then had to identify the direction of the middle stimulus. The test assesses the ability to selectively pay attention to stimuli while inhibiting focus on irrelevant information (Zelazo et al., 2013). On the DCCST, which requires inhibitory control and mental flexibility, participants were first instructed to focus on the color or shape of a forthcoming image and then shown a reference image. Depending on the prompt, participants had to quickly match one of two test images, varying in both color and shape, to the reference image (Zelazo et al., 2013). On the PSMT, participants were presented with a series of pictorial scenes presented in a particular order, and then had to identify the order in which the scenes were presented. The test evaluates learning and immediate retrieval of information (Bauer et al., 2013). For each subtest of the NIHTB-CB, we derived age-corrected standard scores (M = 100, SD = 15).

NIHTB Emotion Battery

The NIHTB-EB includes measures of negative affect, psychosocial well-being, stress and self-efficacy, and social relationships (Salsman et al., 2013). Specifically, there are three scales assessing negative affect or experiences of unpleasant or distressing emotions (i.e., Anger, Fear, and Sadness), two scales assessing psychological well-being or feelings of pleasure and contentment (i.e., Positive Affect and General Life Satisfaction), two scales assessing stress and self-efficacy or one’s perception of everyday experiences and ability to respond to challenging events (i.e., Perceived Stress and Self-Efficacy), and five scales assessing social relationships, with a focus on one’s perception of the availability and quality of those relationships (i.e., Emotional Support, Loneliness, Friendship, Perceived Hostility, and Perceived Rejection). Each of the scales of the NIHTB-EB is completed on a 5-point Likert scale. While some of the scales assess 8- to 17-year-olds, others have distinct forms for 8- to 12-year-olds and 13- to 17-year-olds, with developmentally appropriate wording and item banks. All of the scales for 8- to 12-year-olds and the majority for 13- to 17-year-olds used a fixed number of items; three scales for 13- to 17-year-olds (i.e., Positive Affect, General Life Satisfaction, and Self-Efficacy) used computer-adaptive testing (CAT) methods, in which items are selected from a bank based on participants’ progressive responses. For each scale, we derived fully-corrected t-scores (M = 50, SD = 10).

Clinician-Issued Battery and Rating Scales

For analyses examining convergent and concurrent validity with the NIHTB-CB, participants completed a clinician-issued battery of neuropsychological tests as part of a larger study. They completed the Wechsler Abbreviated Scale of Intelligence, 2^nd edition (WASI-II), a brief assessment of intellectual functioning comprised of four subtests, which provides composite scores of Verbal Reasoning, Fluid Reasoning, and Full Scale IQ. The Vocabulary subtest of the WASI-II assesses crystallized knowledge of vocabulary terms, and can be used in a similar manner as word reading tests to estimate functioning (Bright & van der Linde, 2020). To assess receptive language, participants up to age 16 completed the Comprehension of Instructions subtest from the Neuropsychology Assessment, 2^nd edition (NEPSY-2), in which they were asked to listen to oral instructions of increasing syntactic complexity and point to appropriate stimuli provided. Participants up to age 16 furthermore completed subtests from the Working Memory Index (Digit Span and Letter-Number Sequencing) and Processing Speed Index (Coding and Symbol Search) of the Wechsler Intelligence Scale for Children, 4^th edition (WISC-IV). Three tests of executive functioning from the Delis-Kaplan Executive Function Scale (D-KEFS) were administered to participants above age 8. Specifically, participants completed: the Color-Word Interference Test (CWIT), a variation of the classic Stroop test that requires inhibition of automatic responses when naming stimuli color or word; the Trail Making Test (TMT), which includes a trial of mental flexibility asking individuals to quickly switch between connecting circles containing numbers and letters in order; and, the Verbal Fluency Test (VFT), which includes a trial of mental flexibility asking individuals quickly switch between saying words in different categories. Lastly, participants completed the Design Memory subtest of the Wide Range Assessment of Memory and Learning, 2^nd edition (WRAML-2), a measure of visual learning and memory.

Participants and their parents filled out the Pediatric Quality of Life Inventory (PedsQL), rating scales measuring children’s overall adjustment as well as their adjustment in physical, emotional, school and social domains (with the later three also summarized to index psychosocial adjustment). Parents also completed ratings scales assessing children’s day-to-day adjustment across multiple domains. Parents completed the Adaptive Behavior Assessment System, 2^nd edition (ABAS-II), a questionnaire designed to provide estimates of adaptive behavior, including a measure of overall adaptive skill as well as performance in conceptual, social, and practical domains. Parent-ratings were also collected on the Behavior Assessment System for Children, 2^nd edition (BASC-2), which assesses multiple areas of emotional, behavioral, and adaptive functioning.

Group Differences

Participants with CHD performed significantly poorer than healthy peers on the NIHTB-CB (ps ≤ 0.045), with exception of the LSWMT (Table 1). Still, children and adolescents with CHD generally performed within normal limits. Similarly, on the clinician-issued battery of cognitive tests, participants with CHD tended to perform more poorly than healthy controls but also tended to perform within normal limits (for the comparison data, see Supplemental Table 2). Of note, the overall cognitive level of healthy controls differed from the population mean based on both the NIHTB-CB Total Cognition (t = 5.78, p < 0.001) and the WASI-II Full Scale IQ (t = 6.26, p < 0.001). Similarly, the overall cognitive level of youths with CHD was higher than has been reported in past work (Feldman et al. 2021), suggesting that we captured typical group differences even if participants in both groups were slightly higher functioning than would be expected.

Developmental Trends

Age was positively associated with improved performance on all NIHTB-CB subtests and composite scores, both among participants with CHD (rs ≥ 0.547, ps < 0.001) and healthy controls (rs ≥ 0.400, ps < 0.001). Age effects did not significant differ between participants with CHD and healthy controls, based on Fisher’s r-to-z transformations (zs ≤ |1.71|, ps > 0.08).

Convergent Validity

As shown in Table 2, composite scores on the NIHTB-CB were moderately correlated with comparable composites from the WASI-II for both participants with CHD and typically developing peers, with no significant group differences. For the most part, subtests from the NIHTB-CB were significantly correlated with neuropsychological measures assessing similar constructs across groups, with small-to-medium effect sizes. That being said, the correlations between the NIHTB-CB DCCST and both the D-KEFS VFT Switching and the D-KEFS TMT Number Letter were not significant for either group, likely reflecting that the comparison measures do not sufficiently assess the same construct as the NIHTB-CB DCCST. Moreover, the positive correlation between the NIHTB-CB FIC+AT and the D-KEFS CWIT Inhibition was only significant among participants with CHD; however, the difference between the groups was not statistically significant. The positive correlation between the NIHTB-CB Oral Reading Test and WASI-II Vocabulary was only significant among healthy controls, and this represented the only statistically significant difference between participants with CHD and healthy controls.

Concurrent Validity

When considering whether scores from the NIHTB-CB were associated with measures of day-to-day functioning in a predictable fashion (i.e., assessing concurrent validity), we limited our analyses to composite scores of the NIHTB-CB. As shown in Table 3, for children and adolescents with CHD, better fluid cognition on the NIHTB-CB was associated with fewer externalizing and behavior problems, as rated by parents on the BASC-2. Better crystalized cognition on the NIHTB-CB was associated with better adaptive functioning across composite scores from the ABAS-II and BASC-2 as well as fewer behavior problems as indexed on the BASC-2. Better overall cognition on the NIHTB-CB was associated with better global and practice adaptive functioning on the ABAS-II and BASC-2 as well as fewer internalizing, externalizing, and behavior problems on the BASC-2. Although correlations were not significant for healthy control, there were no significant differences between participants with CHD and healthy controls for any of the analyses.

Group Differences

As depicted in Table 1, there were no significant differences between participants with CHD and typically developing peers for any of the scales from the NIHTH-EB. Overall, both groups of children and adolescents endorsed generally healthy socioemotional functioning.

Confirmatory Factor Analysis

As detailed in Table 4, there was adequate construct coherence for questionnaires from the NIHTB-EB among healthy control, based on the SRMR and CFI from CFA models, echoing studies on the development of the tools (Salsman et al., 2013). Still, the SRMR and CFI only trended towards guidelines of acceptable fit for certain scales (i.e., Sadness, Positive Affect for Ages 8-12, and Perceived Stress for Ages 13-17) and suggested poor fit for one scale (i.e., Positive Affect for Ages 13-17). By comparison, for participants with CHD, the majority of scales did not yield CFA models with SRMR and CFI within guidelines. Even if fit indices often trended towards acceptable ranges, they did not for Positive Affect for Ages 8-12 and for Ages 13-17 and Self-Efficacy for Ages 8-12 and for Ages 13-17. Of note, CFA models across the groups typically yielded RMSEA suggestive of poor fit. However, our small sample sizes likely inflated the RMSEA, making it a less useful measures in evaluating model fit (Kenny, Kaniskan, & McCoach, 2014).

Internal Consistency

As shown in Table 4, analyses using Cronbach’s α revealed acceptable internal consistency between fixed form items among participants with CHD (α ≥ 0.713), similar to healthy controls (α ≥ 0.771), with one exception. Among healthy controls, but not those with CHD, Perceived Stress for Ages 13-17 demonstrated questionable internal consistency (α = 0.606). Similarly, for CAT forms, marginal reliability coefficients from IRT models were suggestive of acceptable internal consistency among both participants with CHD and healthy controls (≥ 0.842).

Convergent Validity

When considering the convergent validity of the self-completed questionnaires from the NIHTB-EB, we focused our analyses on self-reports from the PedsQL. Although research suggests that parents can view their children’s functioning in a divergent manner than children do (Jackson et al., 2015), we also considered parent-reports from the PedsQL, given that we were limited in the number of participants who completed the rating scale. We conducted exploratory analyses using relevant subscales from the parent-completed ABAS-II and BASC-2. As a similar pattern of findings emerged as with the parent-completed PedsQL, we limit our discussion to the PedsQL.

As shown in Table 5, correlations between the scales of the NIHTB-EB and scores from the PedsQL were largely within expectable directions. It should be noted, though, that not all effects, even those with medium effect sizes were significant, as a function of sample size. Importantly, there were no significant group differences, with one exception. Healthy controls who endorsed greater perceived rejection on the NIHTB-EB were rated by their parents as having poorer social functioning on the PedsQL, but the same was not true for children and adolescents with CHD.