Multiple Sclerosis Cognitive Scale (MSCS): A Brief Psychometrically-Robust Metric of Patient-Reported Cognitive Difficulty ========================================================================================================================== * James F. Sumowski * Joshua Sandry ## ABSTRACT **Background** Cognitive decline is common in multiple sclerosis (MS), but existing metrics of patient-reported cognitive difficulties are lengthy, lack psychometric rigor, and/or fail to query expressive language deficits prevalent in MS. **Objective** To develop the Multiple Sclerosis Cognitive Scale (MSCS) as a brief psychometrically-robust metric of patient-reported cognitive deficits. **Method** Exploratory factor analysis (EFA) was conducted on 20 items of the Perceived Deficits Questionnaire (PDQ) plus five newly developed language questions in a large sample of patients with MS and matched respondents without neurologic disease. Confirmatory principal components analysis (PCA) in an independent sample assessed the EFA factor structure. Reliability of the new scale and subscales was assessed, and we evaluated the relationship between the new scale and objective cognitive impairment. **Results** EFA in patients (n=502) and controls (n=350), item analyses, and confirmatory PCA in an independent sample of patients (n=361) and controls (n=150) supported construction of an eight-item scale with four 2-item subscales assessing Executive / Speed, Working Memory, Expressive Language, and Episodic Memory. Internal consistency was excellent for the total MSCS (α=0.93) and good for each subscale (Executive / Speed, α=0.85; Working Memory, α=0.83, Expressive Language, α=0.87; Episodic Memory, α=0.85). There was a medium-sized relationship between objective cognitive impairment and MSCS scores (η2 [95%CI] = 0.06 [0.01, 0.13], but cognitive impairment was not related to the traditional PDQ (0.01 [0.00, 0.06]). **Conclusion** The MSCS is supported as a brief, psychometrically-robust, reliable, and valid metric of patient-reported cognitive deficits in MS. The MSCS holds promise for improving assessment of MS cognitive dysfunction in clinical and research settings. ## INTRODUCTION Cognitive decline is common in multiple sclerosis (MS);(1) it is therefore important to have psychometrically validated, clinically-feasible patient-report tools to screen for cognitive deficits. One option is the Perceived Deficits Questionnaire (PDQ),(2) which is a 20-item scale developed in 1990 with four a priori subscales (attention/concentration, planning/organization, retrospective memory, prospective memory). The PDQ has limitations: subscales were not validated by factor analytic studies, there are no questions about expressive language deficits although word-finding difficulty is frequently reported by patients,(3) and the relatively long length of the PDQ reduces clinical feasibility due to patient burden, especially when included within a wider collection of questionnaires (e.g., fatigue, mood, physical disability). Another option is the MS Neuropsychological Questionnaire (MSNQ),(4) which is a 15-item scale developed in 2003 without cognitive subscales; items focus heavily on attention / executive function and memory without assessment of word-finding difficulty. It is also notable that the MSNQ and PDQ were developed over two and three decades ago, respectively. Others have more recently recommended Neuro-QoL short forms including a brief questionnaire assessing communication difficulties;(5) however, even these lack questions on word-finding difficulty. The current study aims to develop a brief, reliable, clinically-useful measure of patient-reported cognitive difficulties with cognitive subscales validated with factor analytic techniques. ## METHODS ### Sample The Corinne Goldsmith Dickinson Center for Multiple Sclerosis at Mount Sinai Hospital is a tertiary care center with a catchment area encompassing the racially/ethnically diverse New York Metropolitan Area. In 2018, we established a clinic aiming to perform cognitive screenings for all patients at our center; the 20-item PDQ plus five language questions was completed by patients from August 2018 through August 2021. We performed an IRB approved retrospective chart review of clinical and patient-reported cognitive data from all patients aged 18 to 65 years and diagnosed with relapse-onset MS from 1995 onward who completed cognitive screenings. Self-reported cognitive difficulty was also collected from demographically-matched persons without neurologic conditions through an IRB-exempt anonymous electronic capture questionnaire. ### PDQ plus Language Questions The PDQ is a 20-item self-report inventory assessing frequency of cognitive difficulties as never (0), rarely (1), sometimes (2), fairly often (3), or very often (4). The PDQ has four *a priori* subscales (with five items each) assessing attention/concentration, planning/organization, retrospective memory, and prospective memory, but there is no assessment of language difficulty. Our development of five language questions to add to the PDQ was informed by clinical experience with patients and confirmed by retrospective chart review of self-report remote electronic capture (REDCap) questionnaire in consecutive patients (meeting aforementioned inclusion criteria) during a 12-month period. Patients were asked “Do you have any concerns about your current cognition?” If yes, an open field question displayed: “In a few words, please describe the cognitive problem(s) that you experience.” These questions were completed prior to completion of the PDQ to avoid biasing results. Three independent raters (see acknowledgements) reviewed all free text responses to identify presence or absence of self-reported language difficulty, and then to categorize difficulties into like categories. ### Statistical Approach Exploratory factor analysis was performed on responses to the 25 questions by patients and controls using R version 4.2.1 and the psych package.(6) The goal was to develop an empirically-derived brief cognitive screener; as such, we identified the two items with the highest factor loadings and highest internal consistency (Cronbach’s alpha) for each factor, which were selected for the new brief screener. Next, confirmatory principal components analysis was performed for selected items in an independent sample of patients and controls. ## RESULTS ### Sample Data on the 25-item self-report cognitive questionnaire were captured from 502 persons with MS and 350 demographically-matched control respondents for the exploratory factor analysis (Table 1). View this table: [Table 1.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T1) Table 1. Sample Characteristics ### Language Questions Data were captured from a subset of 319 consecutive patients with MS for validation of language questions. Of 319 patients, about half (51%, n=163) endorsed “yes” to having concerns about cognition; free text responses were reviewed by three independent raters to identify presence or absence of self-reported language difficulty (interrater agreement was excellent, Kappa [95%CI] of 0.85 [0.76, 0.93]). Language deficits were considered present if all three raters (n=54) or two of three raters (n=10) coded it present; language difficulty was reported by 39% of patients endorsing concerns about cognition (n=64 of 163), and 20% of all patients regardless of endorsement of cognitive concerns (n=64 of 319). Analysis of the 64 responses identified (a) word-finding difficulty (n=46) with descriptions of trouble retrieving known words (i.e., “tip of the tongue” phenomenon), (b) difficulty clearly expressing thoughts (n=13), (c) using the wrong word or misspeaking (n=10), (d) difficulties comprehending text or discourse (n=6), and (e) other and/or vague responses (n=6; e.g., “speech problems,” misspelling). Analyses support use of the five language questions (last five items in Table 2). View this table: [Table 2.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T2) Table 2. Results of Exploratory Factor Analysis of 25 Self-Reported Cognitive Deficits The 25 items are from the 20-item PDQ plus five new language items (in italics below). ### Exploratory Factor Analysis (EFA) Responses to the 25 questions by 852 consecutive respondents (502 patients, 350 controls) were analyzed with EFA using R version 4.2.1 and the psych package. The Kaiser-Meyer-Olkin measure of sampling adequacy was .98 suggesting excellent factorability. Results from the parallel analysis, in concordance with the scree plot, suggested that a five-factor solution with oblimin rotation has excellent fit (RMSEA = .05, TLI = .962; Table 2, Figure 1). Of the four items with highest loadings for each factor, we identified the two with the highest internal consistency (Cronbach’s alpha). For factor A, internal consistency was good for “get started when lots to do” and “take long time to finish things” (α=0.88). For factor B, internal consistency was acceptable for “forget to take medication, etc.” and “forget meetings, appointments” (α=0.79). For factor C, internal consistency was good for “word on ‘tip of tongue’” and “clearly expressing thoughts” (α=0.85). Factor D consisted of two items with good internal consistency (α=0.85). For factor E, internal consistency was good for “forget why entered room” and “losing train of thought” (α=0.85). Internal consistency was good for all two-item pairs across factors except factor B. Further inspection revealed possible floor effects for all four items with highest loadings for factor B, with < 10% of all respondents endorsing “fairly often” or “very often.” These items are four of the five items of the PDQ prospective memory subscale; means for this scale were much lower than all other PDQ subscales in the original publication.(2) Only one of the other 21 questions showed a possible floor effect (“follow complex conversations”). To derive a brief scale with the best reliability and clinical relevance, we excluded factor B. EFA with the remaining eight items yielded four factors (Table 3, Figure 2), which are best characterized as Executive / Speed, Episodic Memory, Working Memory, and Expressive Language. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/05/01/2024.04.29.24306587/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/F1) Figure 1. Exploratory Factor Analysis of 25 Self-Reported Cognitive Deficits View this table: [Table 3.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T3) Table 3. Results of Exploratory Factor Analysis of Eight MS Cognitive Scale (MSCS) Items ![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/05/01/2024.04.29.24306587/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/F2) Figure 2. Exploratory Factor Analysis of the Eight MSCS Items. ### Confirmatory Principal Components Analyses (PCA) Confirmatory PCAs (four components, oblimin rotation) with the eight selected items were performed in an independent sample combining a research sample of 167 persons with relapsing-remitting MS who completed the full 25-item survey, a clinical sample of 194 patients with MS who completed the brief scale, and a sample of 150 respondents without neurologic conditions who completed the brief scale. As shown (Table 4), confirmatory PCA in this independent sample supported the factor structure of the eight-item MSCS, which was also shown when performing separate PCAs for all patients (n=863) and all controls (n=500; Table 5). View this table: [Table 4.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T4) Table 4. Results of Confirmatory Principal Component Analyses in Independent Sample View this table: [Table 5.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T5) Table 5. Results of Confirmatory Principal Component Analyses Separately for All Patients and All Controls ### Reliability Internal consistency (Cronbach’s alpha) of the eight-item scale among patients who completed the brief form (n=194) was excellent for the total MSCS (α=0.93) and good for each subscale (Executive / Speed, α=0.85; Episodic Memory, α=0.85; Working Memory, α=0.83, Expressive Language, α=0.87). Test-retest reliability was assessed with intraclass correlation coefficients (ICC; two-way mixed ANOVA; absolute agreement(7)) for the brief form completed twice by 62 patients with MS, with a one-year interval between assessments; reliability was good for the total MSCS (ICC [95% CI], 0.82 [0.72, 0.89]) and for each subscale (ICCs 0.74 to 0.81; Table 6). This is likely an underestimate of the actual test-retest reliability given the long one-year interval between assessments. View this table: [Table 6.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T6) Table 6. Test-Retest Reliabilities of MSCS Subscales in Patients completing the brief form at two time points (n=62) ICC [95%CI] with two-way mixed analysis of variance model with interaction for the absolute agreement between single scores. ICC for the total MSCS score was 0.82 [0.72, 0.89]. ### Link to Objective Cognitive Performance A brief screening battery adopted for MS (BICAMS)(8) consists of a high-sensitivity information processing task (Symbol Digit Modalities Test, SDMT)(9) and measures of word-list learning and object-location memory. Our modified version of this battery includes SDMT, word-list total learning on the Hopkins Verbal Learning Test, Revised (HVLT-R),(10) and object-location memory on CANTAB Paired Associate Learning (PAL).(11) Task performance data were available for 502 patients who completed the full 20-item PDQ and 194 patients who completed the MSCS. Raw scores were converted to age-adjusted norm-referenced z-scores relative to each test’s healthy standardization sample, which was then used to characterize impairment for each task as performance ≤1.5 standard deviations below normal (z-score ≤ -1.5). Patients were categorized as having impairment on 0, 1, or 2+ tests. We then matched patients from the larger PDQ sample to the smaller MSCS sample for age, sex, race/ethnicity, education, MS phenotype, time since diagnosis, and mood (MHI-5), resulting in extremely well-matched samples of (a) 194 patients who completed the full 20-item PDQ and (b) 194 patients who completed the MSCS (Table 7). One-way ANOVAs tested differences in the PDQ (mean of 20 items) and the MSCS (mean of eight items) across patients with impairment on 0, 1, or 2+ tests. As shown (Figure 3), MSCS differed across levels of cognitive impairment (F[2, 193]=5.85, p=0.003; η2 [95%CI] = 0.06 [0.01, 0.13]) whereby patient-reported difficulty was worse among patients with 2+ impaired tests than those with ≤1 impaired test. In contrast, there was no difference in PDQ across levels of cognitive impairment (F[2, 193]=1.34, p=0.265; η2 [95%CI] = 0.01 [0.00, 0.06]). One-way ANOVAs were repeated after adjusting MSCS and PDQ for mood (MHI-5, using GLM). Again, as shown (Figure 3), there were differences in MSCS across levels of cognitive impairment F[2, 193]= 3.83, p=0.023; η2 [95%CI] = 0.04 [0.00, 0.10]), but PDQ did not differ across levels of impairment (F[2, 193]=0.15, p=0.864; η2 [95%CI] = 0.00 [0.00, 0.02]). View this table: [Table 7.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T7) Table 7. Matched samples of patients with MS who completed the PDQ or the MSCS ![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/05/01/2024.04.29.24306587/F3.medium.gif) [Figure 3.](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/F3) Figure 3. Differences in Patient-Reported Cognitive Difficulty across levels of Cognitive Impairment in Matched Samples completing the PDQ or MSCS Results are presented for MSCS and PDQ scores before (left) and after (right) adjusting scores for mood (MHI-5). Effect sizes (Cohen’s D [95%CI]) are presented for comparisons across levels of cognitive impairment. ## DISCUSSION We report good reliability and validity for the Multiple Sclerosis Cognitive Scale (MSCS; see Appendix), a new eight-item patient-report cognitive questionnaire with four factor analytically derived subscales (executive / speed, working memory, expressive language, episodic memory). MSCS showed a medium-sized relationship to objective cognitive impairment, and remained reliably different even when adjusting for mood. In contrast, the traditional PDQ was unrelated to objective impairment with and without adjusting for mood, despite having 2.5 times as many items as the MSCS. It may be that patients respond more thoughtfully when there are fewer items, and that MSCS items better represent the cognitive problems experienced by persons living with MS, especially given assessment of expressive language difficulty (which is missing from existing patient-report cognitive scales). The MSCS holds promise as a brief, reliable, psychometrically-robust self-report scale with good links to objective cognitive impairment. ## Data Availability All data produced in the present study are available upon reasonable request to the authors ## Acknowledgements The authors thank the patients, faculty, and staff of the Corinne Goldsmith Dickinson Center for Multiple Sclerosis. Special thanks to Jordyn Anderson, PsyD, Hanaan Bing-Canar, PhD, and Emily Dvorak for their work as independent raters of responses. ## APPENDIX Multiple Sclerosis Cognitive Scale (MSCS) **Please check the box to indicate how frequently during the past month you experienced:** View this table: [Table8](http://medrxiv.org/content/early/2024/05/01/2024.04.29.24306587/T8) ## Footnotes * Funding Statement: This project was funded in part by the National Institute of Child Health and Development (NICHD) within the National Institutes of Health (NIH; R01 HD082176 to JFS). * Disclosures: The authors have no conflicts of interest related to the current project. * Received April 29, 2024. * Revision received April 29, 2024. * Accepted May 1, 2024. * © 2024, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## REFERENCES 1. 1.Sumowski JF, Benedict R, Enzinger C, Filippi M, Geurts JJ, Hamalainen P, et al. Cognition in multiple sclerosis: State of the field and priorities for the future. Neurology. 2018 Feb 6;90(6):278–88. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1212/WNL.0000000000004977&link_type=DOI) 2. 2.Sullivan M, Edgley K, Dehoux E. A survey of multiple sclerosis: I. Perceived cognitive problems and compensatory strategy use. Canadian Journal of Rehabilitation. 1990;4:99–105. 3. 3.Brandstadter R, Fabian M, Leavitt VM, Krieger S, Yeshokumar A, Katz Sand I, et al. Word-finding difficulty is a prevalent disease-related deficit in early multiple sclerosis. Mult Scler. 2020 Nov;26(13):1752–64. 4. 4.Benedict RHB, Munschauer F, Linn R, Miller C, Murphy E, Foley F, et al. Screening for multiple sclerosis cognitive impairment using a self-administered 15-item questionnaire. Mult Scler. 2003 Feb;9(1):95–101. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1191/1352458503ms861oa&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12617275&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F05%2F01%2F2024.04.29.24306587.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000181156300016&link_type=ISI) 5. 5.Medina LD, Torres S, Alvarez E, Valdez B, Nair K V. Patient-reported outcomes in multiple sclerosis: Validation of the Quality of Life in Neurological Disorders (Neuro-QoLTM) short forms. Mult Scler J Exp Transl Clin. 2019;5(4):2055217319885986. 6. 6.Revelle W, Revelle M. Package “psych.” The comprehensive R archive network. 2015;337(338). 7. 7.Qin S, Nelson L, McLeod L, Eremenco S, Coons SJ. Assessing test-retest reliability of patient-reported outcome measures using intraclass correlation coefficients: recommendations for selecting and documenting the analytical formula. Qual Life Res. 2019 Apr;28(4):1029–33. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11136-018-2076-0&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F05%2F01%2F2024.04.29.24306587.atom) 8. 8.Langdon DW, Amato MP, Boringa J, Brochet B, Foley F, Fredrikson S, et al. Recommendations for a Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS). Mult Scler. 2012 Jun;18(6):891–8. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1352458511431076&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22190573&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F05%2F01%2F2024.04.29.24306587.atom) 9. 9.Smith A. Symbol digit modalities test (SDMT) manual (revised) . Los Angeles: Western Psychological Services; 1982. 10. 10.Brandt J, Benedict R. Hopkins Verbal Learning Test, Revised. Psychological Assessment Resources, Inc.; 2001. 11. 11.Barnett JH, Blackwell AD, Sahakian BJ, Robbins TW. The Paired Associates Learning (PAL) Test: 30 Years of CANTAB Translational Neuroscience from Laboratory to Bedside in Dementia Research. Curr Top Behav Neurosci. 2016;28:449–74. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F05%2F01%2F2024.04.29.24306587.atom)