ABSTRACT
Background Cognitive decline is common in multiple sclerosis (MS), but existing metrics of patient-reported cognitive difficulties are lengthy, lack psychometric rigor, and/or fail to query expressive language deficits prevalent in MS.
Objective To develop the Multiple Sclerosis Cognitive Scale (MSCS) as a brief psychometrically-robust metric of patient-reported cognitive deficits.
Method Exploratory factor analysis (EFA) was conducted on 20 items of the Perceived Deficits Questionnaire (PDQ) plus five newly developed language questions in a large sample of patients with MS and matched respondents without neurologic disease. Confirmatory principal components analysis (PCA) in an independent sample assessed the EFA factor structure. Reliability of the new scale and subscales was assessed, and we evaluated the relationship between the new scale and objective cognitive impairment.
Results EFA in patients (n=502) and controls (n=350), item analyses, and confirmatory PCA in an independent sample of patients (n=361) and controls (n=150) supported construction of an eight-item scale with four 2-item subscales assessing Executive / Speed, Working Memory, Expressive Language, and Episodic Memory. Internal consistency was excellent for the total MSCS (α=0.93) and good for each subscale (Executive / Speed, α=0.85; Working Memory, α=0.83, Expressive Language, α=0.87; Episodic Memory, α=0.85). There was a medium-sized relationship between objective cognitive impairment and MSCS scores (η2 [95%CI] = 0.06 [0.01, 0.13], but cognitive impairment was not related to the traditional PDQ (0.01 [0.00, 0.06]).
Conclusion The MSCS is supported as a brief, psychometrically-robust, reliable, and valid metric of patient-reported cognitive deficits in MS. The MSCS holds promise for improving assessment of MS cognitive dysfunction in clinical and research settings.
INTRODUCTION
Cognitive decline is common in multiple sclerosis (MS);(1) it is therefore important to have psychometrically validated, clinically-feasible patient-report tools to screen for cognitive deficits. One option is the Perceived Deficits Questionnaire (PDQ),(2) which is a 20-item scale developed in 1990 with four a priori subscales (attention/concentration, planning/organization, retrospective memory, prospective memory). The PDQ has limitations: subscales were not validated by factor analytic studies, there are no questions about expressive language deficits although word-finding difficulty is frequently reported by patients,(3) and the relatively long length of the PDQ reduces clinical feasibility due to patient burden, especially when included within a wider collection of questionnaires (e.g., fatigue, mood, physical disability). Another option is the MS Neuropsychological Questionnaire (MSNQ),(4) which is a 15-item scale developed in 2003 without cognitive subscales; items focus heavily on attention / executive function and memory without assessment of word-finding difficulty. It is also notable that the MSNQ and PDQ were developed over two and three decades ago, respectively. Others have more recently recommended Neuro-QoL short forms including a brief questionnaire assessing communication difficulties;(5) however, even these lack questions on word-finding difficulty. The current study aims to develop a brief, reliable, clinically-useful measure of patient-reported cognitive difficulties with cognitive subscales validated with factor analytic techniques.
METHODS
Sample
The Corinne Goldsmith Dickinson Center for Multiple Sclerosis at Mount Sinai Hospital is a tertiary care center with a catchment area encompassing the racially/ethnically diverse New York Metropolitan Area. In 2018, we established a clinic aiming to perform cognitive screenings for all patients at our center; the 20-item PDQ plus five language questions was completed by patients from August 2018 through August 2021. We performed an IRB approved retrospective chart review of clinical and patient-reported cognitive data from all patients aged 18 to 65 years and diagnosed with relapse-onset MS from 1995 onward who completed cognitive screenings. Self-reported cognitive difficulty was also collected from demographically-matched persons without neurologic conditions through an IRB-exempt anonymous electronic capture questionnaire.
PDQ plus Language Questions
The PDQ is a 20-item self-report inventory assessing frequency of cognitive difficulties as never (0), rarely (1), sometimes (2), fairly often (3), or very often (4). The PDQ has four a priori subscales (with five items each) assessing attention/concentration, planning/organization, retrospective memory, and prospective memory, but there is no assessment of language difficulty. Our development of five language questions to add to the PDQ was informed by clinical experience with patients and confirmed by retrospective chart review of self-report remote electronic capture (REDCap) questionnaire in consecutive patients (meeting aforementioned inclusion criteria) during a 12-month period. Patients were asked “Do you have any concerns about your current cognition?” If yes, an open field question displayed: “In a few words, please describe the cognitive problem(s) that you experience.” These questions were completed prior to completion of the PDQ to avoid biasing results. Three independent raters (see acknowledgements) reviewed all free text responses to identify presence or absence of self-reported language difficulty, and then to categorize difficulties into like categories.
Statistical Approach
Exploratory factor analysis was performed on responses to the 25 questions by patients and controls using R version 4.2.1 and the psych package.(6) The goal was to develop an empirically-derived brief cognitive screener; as such, we identified the two items with the highest factor loadings and highest internal consistency (Cronbach’s alpha) for each factor, which were selected for the new brief screener. Next, confirmatory principal components analysis was performed for selected items in an independent sample of patients and controls.
RESULTS
Sample
Data on the 25-item self-report cognitive questionnaire were captured from 502 persons with MS and 350 demographically-matched control respondents for the exploratory factor analysis (Table 1).
Language Questions
Data were captured from a subset of 319 consecutive patients with MS for validation of language questions. Of 319 patients, about half (51%, n=163) endorsed “yes” to having concerns about cognition; free text responses were reviewed by three independent raters to identify presence or absence of self-reported language difficulty (interrater agreement was excellent, Kappa [95%CI] of 0.85 [0.76, 0.93]). Language deficits were considered present if all three raters (n=54) or two of three raters (n=10) coded it present; language difficulty was reported by 39% of patients endorsing concerns about cognition (n=64 of 163), and 20% of all patients regardless of endorsement of cognitive concerns (n=64 of 319). Analysis of the 64 responses identified (a) word-finding difficulty (n=46) with descriptions of trouble retrieving known words (i.e., “tip of the tongue” phenomenon), (b) difficulty clearly expressing thoughts (n=13), (c) using the wrong word or misspeaking (n=10), (d) difficulties comprehending text or discourse (n=6), and (e) other and/or vague responses (n=6; e.g., “speech problems,” misspelling). Analyses support use of the five language questions (last five items in Table 2).
The 25 items are from the 20-item PDQ plus five new language items (in italics below).
Exploratory Factor Analysis (EFA)
Responses to the 25 questions by 852 consecutive respondents (502 patients, 350 controls) were analyzed with EFA using R version 4.2.1 and the psych package. The Kaiser-Meyer-Olkin measure of sampling adequacy was .98 suggesting excellent factorability. Results from the parallel analysis, in concordance with the scree plot, suggested that a five-factor solution with oblimin rotation has excellent fit (RMSEA = .05, TLI = .962; Table 2, Figure 1). Of the four items with highest loadings for each factor, we identified the two with the highest internal consistency (Cronbach’s alpha). For factor A, internal consistency was good for “get started when lots to do” and “take long time to finish things” (α=0.88). For factor B, internal consistency was acceptable for “forget to take medication, etc.” and “forget meetings, appointments” (α=0.79). For factor C, internal consistency was good for “word on ‘tip of tongue’” and “clearly expressing thoughts” (α=0.85). Factor D consisted of two items with good internal consistency (α=0.85). For factor E, internal consistency was good for “forget why entered room” and “losing train of thought” (α=0.85). Internal consistency was good for all two-item pairs across factors except factor B. Further inspection revealed possible floor effects for all four items with highest loadings for factor B, with < 10% of all respondents endorsing “fairly often” or “very often.” These items are four of the five items of the PDQ prospective memory subscale; means for this scale were much lower than all other PDQ subscales in the original publication.(2) Only one of the other 21 questions showed a possible floor effect (“follow complex conversations”). To derive a brief scale with the best reliability and clinical relevance, we excluded factor B. EFA with the remaining eight items yielded four factors (Table 3, Figure 2), which are best characterized as Executive / Speed, Episodic Memory, Working Memory, and Expressive Language.
Confirmatory Principal Components Analyses (PCA)
Confirmatory PCAs (four components, oblimin rotation) with the eight selected items were performed in an independent sample combining a research sample of 167 persons with relapsing-remitting MS who completed the full 25-item survey, a clinical sample of 194 patients with MS who completed the brief scale, and a sample of 150 respondents without neurologic conditions who completed the brief scale. As shown (Table 4), confirmatory PCA in this independent sample supported the factor structure of the eight-item MSCS, which was also shown when performing separate PCAs for all patients (n=863) and all controls (n=500; Table 5).
Reliability
Internal consistency (Cronbach’s alpha) of the eight-item scale among patients who completed the brief form (n=194) was excellent for the total MSCS (α=0.93) and good for each subscale (Executive / Speed, α=0.85; Episodic Memory, α=0.85; Working Memory, α=0.83, Expressive Language, α=0.87). Test-retest reliability was assessed with intraclass correlation coefficients (ICC; two-way mixed ANOVA; absolute agreement(7)) for the brief form completed twice by 62 patients with MS, with a one-year interval between assessments; reliability was good for the total MSCS (ICC [95% CI], 0.82 [0.72, 0.89]) and for each subscale (ICCs 0.74 to 0.81; Table 6). This is likely an underestimate of the actual test-retest reliability given the long one-year interval between assessments.
ICC [95%CI] with two-way mixed analysis of variance model with interaction for the absolute agreement between single scores. ICC for the total MSCS score was 0.82 [0.72, 0.89].
Link to Objective Cognitive Performance
A brief screening battery adopted for MS (BICAMS)(8) consists of a high-sensitivity information processing task (Symbol Digit Modalities Test, SDMT)(9) and measures of word-list learning and object-location memory. Our modified version of this battery includes SDMT, word-list total learning on the Hopkins Verbal Learning Test, Revised (HVLT-R),(10) and object-location memory on CANTAB Paired Associate Learning (PAL).(11) Task performance data were available for 502 patients who completed the full 20-item PDQ and 194 patients who completed the MSCS. Raw scores were converted to age-adjusted norm-referenced z-scores relative to each test’s healthy standardization sample, which was then used to characterize impairment for each task as performance ≤1.5 standard deviations below normal (z-score ≤ -1.5). Patients were categorized as having impairment on 0, 1, or 2+ tests. We then matched patients from the larger PDQ sample to the smaller MSCS sample for age, sex, race/ethnicity, education, MS phenotype, time since diagnosis, and mood (MHI-5), resulting in extremely well-matched samples of (a) 194 patients who completed the full 20-item PDQ and (b) 194 patients who completed the MSCS (Table 7). One-way ANOVAs tested differences in the PDQ (mean of 20 items) and the MSCS (mean of eight items) across patients with impairment on 0, 1, or 2+ tests. As shown (Figure 3), MSCS differed across levels of cognitive impairment (F[2, 193]=5.85, p=0.003; η2 [95%CI] = 0.06 [0.01, 0.13]) whereby patient-reported difficulty was worse among patients with 2+ impaired tests than those with ≤1 impaired test. In contrast, there was no difference in PDQ across levels of cognitive impairment (F[2, 193]=1.34, p=0.265; η2 [95%CI] = 0.01 [0.00, 0.06]). One-way ANOVAs were repeated after adjusting MSCS and PDQ for mood (MHI-5, using GLM). Again, as shown (Figure 3), there were differences in MSCS across levels of cognitive impairment F[2, 193]= 3.83, p=0.023; η2 [95%CI] = 0.04 [0.00, 0.10]), but PDQ did not differ across levels of impairment (F[2, 193]=0.15, p=0.864; η2 [95%CI] = 0.00 [0.00, 0.02]).
Results are presented for MSCS and PDQ scores before (left) and after (right) adjusting scores for mood (MHI-5). Effect sizes (Cohen’s D [95%CI]) are presented for comparisons across levels of cognitive impairment.
DISCUSSION
We report good reliability and validity for the Multiple Sclerosis Cognitive Scale (MSCS; see Appendix), a new eight-item patient-report cognitive questionnaire with four factor analytically derived subscales (executive / speed, working memory, expressive language, episodic memory). MSCS showed a medium-sized relationship to objective cognitive impairment, and remained reliably different even when adjusting for mood. In contrast, the traditional PDQ was unrelated to objective impairment with and without adjusting for mood, despite having 2.5 times as many items as the MSCS. It may be that patients respond more thoughtfully when there are fewer items, and that MSCS items better represent the cognitive problems experienced by persons living with MS, especially given assessment of expressive language difficulty (which is missing from existing patient-report cognitive scales). The MSCS holds promise as a brief, reliable, psychometrically-robust self-report scale with good links to objective cognitive impairment.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
Acknowledgements
The authors thank the patients, faculty, and staff of the Corinne Goldsmith Dickinson Center for Multiple Sclerosis. Special thanks to Jordyn Anderson, PsyD, Hanaan Bing-Canar, PhD, and Emily Dvorak for their work as independent raters of responses.
APPENDIX Multiple Sclerosis Cognitive Scale (MSCS)
Please check the box to indicate how frequently during the past month you experienced:
Footnotes
Funding Statement: This project was funded in part by the National Institute of Child Health and Development (NICHD) within the National Institutes of Health (NIH; R01 HD082176 to JFS).
Disclosures: The authors have no conflicts of interest related to the current project.
