Analytical validation of a multi-protein, serum-based assay for disease activity assessments in multiple sclerosis

2.1 Assay development

Twenty-one biomarkers were selected for inclusion in the custom assay panel based on statistical associations with clinical and radiographic endpoints as demonstrated in feasibility studies for which >1400 proteins were screened using 2 immunoassay platforms (Table S1). These feasibility studies investigated biomarker associations (single-protein and multi-protein) in both cross-sectional and longitudinal samples relative to several radiographic and clinical MS endpoints, including clinically defined relapse versus remission (exacerbation versus quiescence), the presence and count of gadolinium-enhanced lesions on a matched MRI, annualized relapse rate, and Expanded Disability Status Scale. From these studies, the custom panel of 21 proteins was selected with a primary focus on the detection and prediction of disease activity status. The 21 proteins were chosen based on their statistical significance relative to the aforementioned endpoints and with the intent to comprehensively survey the biological pathways, mechanisms, and cell types associated with MS pathophysiology as determined via literature review, protein-protein interaction modeling, gene set enrichment, and spatial expression profiling.[25] Dynamic range of the individual protein assays was considered, as well as the intent to develop a single multi-protein immunoassay panel for which each protein could be measured in an undiluted serum sample. The MSDA Test algorithm consisting of 18 biomarkers included in the panel was finalized in a subsequent clinical validation study for which independent sample sets were analyzed. The final model was trained and validated relative to the presence and count of gadolinium-enhanced lesions.

Serum pools (n=4 were included on all runs during assay discovery and development. They were procured in large volumes, aliquoted, stored at −65°C, and run in triplicate. Serum pools were used solely to assess the analytical performance of the assays and served as process controls to determine acceptability of future analytical runs. The SD of repeated measurements was applied to the expected concentrations. Two assay kit lots of the panel were manufactured for which critical reagents were varied to the extent possible.

2.2 Description of the two-layer stacked classifier algorithm for determination of the overall Disease Activity score

A two-layer, L2-penalized logistic regression stacked classifier model was developed and clinically validated in a separate study that optimized the model’s performance to classify serum samples based on the presence of gadolinium-enhancing lesions (0 lesions or ≥1 lesions) on an MRI administered within 60 days of blood draw.[26] In the first layer of the model, individual protein concentrations in log₁₀ which were demographically corrected for age and sex and LOQ-imputed (referred to as adjusted concentrations) were used as inputs into the four Disease Pathway models (Immunomodulation, Neuroinflammation, Myelin Biology, and Neuroaxonal Integrity). The second layer of the model used the adjusted protein concentrations and the output (eg, the probability) of the Disease Pathway models as meta features to calculate an overall Disease Activity score (File S1, Supporting Information). Thresholds were established, which corresponded to low (1.0□4.0), moderate (4.5□7.0), and high (7.5□10.0) Disease Activity scores. Analytical characterization and validation of the individual biomarkers were factors used to determine inclusion of those biomarkers in the algorithm.

2.3 Incurred sample reanalysis

Incurred sample reanalysis was performed to characterize precision and robustness for the individual biomarkers and the Disease Activity and Disease Pathway scores. Forty-eight individual samples from patients with MS were repeatedly analyzed across 10 plates over ≥5 days with varied equipment, reagents, location, and personnel. Acceptability criteria for individual biomarkers was an average %CV ≤20%, and average SD at all established Disease Activity score levels of ≤1.0 units. The 48 samples broadly represented the expected range of biomarker values and Disease Activity scores in the real-world MS population.

2.4 Assay accuracy, precision, and sensitivity

Accuracy for each analyte was determined by mixing serum samples at different ratios and evaluating the percent recovery of the observed concentration relative to the expected concentration. Sample mixing enabled the accuracy assessment to be performed using endogenous protein versus a recombinant protein source. Expected concentrations were calculated by applying the targeted ratios of unmixed samples. The ratios of sample mixtures with two samples were 25%:75%, 50%:50%, and 75%:25%. The ratios of sample blends for mixtures with four samples were 25%:25%:25%:25% and 40%:10%:40%:10%. Additionally, accuracy was also evaluated for the Disease Pathway and Disease Activity algorithms by correlating observed scores with expected scores using the same sample mixtures created for the individual analyte assessments.

Intra- and inter-assay precision was measured for each analyte. The %CV was determined using serum pools enabling the assessment to be performed using endogenous protein. Serum pools were manufactured to represent patients with shorter and longer MS disease duration, those with inflammatory disease (rheumatoid arthritis), and one healthy control. Acceptability criteria for intra- and inter-assay precision was established as %CV ≤15% and ≤20%, respectively.

Sensitivity was defined as the assay’s ability to accurately and precisely detect low concentrations of a given substance in biological specimens. To establish the ULOQ and LLOQ, a LOQ panel was manufactured during assay development. For each analyte, four levels were targeted near the anticipated upper limit (ULOQ 1‒4) and four levels were targeted near the anticipated lower limit (LLOQ 5‒8). The targeted concentrations were based on expected real-world MS patient sample distributions, the shape of the standard curve, and location of asymptotes. The LOQ panel was run in triplicate over two lots (≥5 runs per lot) and fit to the standard curve. Accuracy, defined as 80‒120% recovery relative to the expected concentration and precision (inter-assay %CV ≤20%), were used to establish the acceptability criteria and determine the LLOQ and ULOQ of each analyte. Additionally, individual LOQs were assessed and established separately for each kit lot. The most conservative LOQ levels with acceptable accuracy and precision parameters for both lots were used to establish the final LLOQ and ULOQ.

Undiluted serum samples were run in the MSDA Test and as a result, no dilution factor was accounted for in the sensitivity analysis. Therefore, the LLOQ and ULOQ define both the analytical measurement range and the reportable range of the assay. Serum samples that recovered either above the ULOQ or below the LLOQ were reported at the established LOQ concentration (referred to as LOQ imputation). MS serum samples were used to establish MS reference ranges for each biomarker. A diverse set of patient samples were used throughout the assay development process and for the analytical validation studies. A total of 1645 samples from nine deeply phenotyped cohorts were analyzed primarily for evaluating associations of biomarkers with MS disease activity and disease progression endpoints. Additional samples from both patients with MS and other disease states were procured for specific analytical characterization experiments. The 1645 samples that were analyzed for the associations of biomarkers with MS endpoints were combined in the subsequent analysis to establish MS reference ranges. These samples were collected both retrospectively and prospectively from nine US and international sites and broadly represent the real-world MS population. The mean ± SD age of these patients at the time of the blood draw was 40.85 ± 11.0 years, with a mean ± SD disease duration of 8.39 ± 8.0 years; 72.8% of the patients were female. For race, the top 3 categories were White (81.4%), unknown/not reported (13.5%), and Black/African American (2.7%). The primary endpoint used to train the finalized MSDA Test algorithm was the presence and count of gadolinium-enhancing lesions on an MRI administered within close proximity to the blood draw. For the 1645 patient samples, 1326 had available gadolinium-positive (Gd+) lesion counts and 53.0% of the patient samples had ≥1 Gd+ lesion. The linear interpolation method was used to establish the 95% interval (2.5th and 97.5th percentiles).[27] The percentile relative to these reference ranges are presented with their protein concentrations.

2.5 Assay interference

Assay interference was defined as the effect of a substance present in the sample altering the correct value of the result or the recovery of samples in the assay. Since patients with MS may be treated with a variety of drugs, potential interference of drugs was tested to determine if their presence would affect measurement of the individual protein biomarkers. Concentrations of common prescriptions, over-the- counter drugs, common MS drugs, and DMTs were spiked into serum samples (Table S2). Concentrations of common prescription and over-the-counter drugs were determined by Sun Diagnostics (New Gloucester, ME, USA) using a commercially available test kit. DMTs were targeted at two times C_max from pharmacokinetic studies, or the highest possible concentration allowable for spiking with the procured interferent stock. Finally, a universal mAb standard was tested at two concentrations (424 and 7.93 µg/mL) to cover the two times C_max of several mAb DMTs. Endogenous substances (hemoglobin, bilirubin, and lipids) and heterophilic antibodies (RF and HAMA) were also measured. For most interferent substances, the acceptability threshold, or median recovery, for the interference assessment was established as 80□120% relative to a corresponding spike control, except for HAMA for which percent recovery of sample mixtures was evaluated (File S2, Supporting Information).

2.6 Diurnal variability

Patient serum samples were collected at days 1□5 and day 12 to characterize biomarker level fluctuations. For each of the six time points per patient, the %CV and the percentage difference of the observed protein concentration relative to the average concentration at all time points were calculated.

2.7 Sample stability

In an initial experiment, stability studies for four serum samples were performed to determine the effect that storage and processing conditions can have in a clinical setting. Stability was assessed at the following four temperatures: −65°C or below (−80°C), −10°C or below (−20°C), 2‒8°C (4°C), and room temperature (18‒25°C) at the following time points: 4 hours (for 4°C and room temperature) and days 1, 3, 7, 14, and 28 (for −20°C, 4°C, and room temperature). The results from −20°C, 4°C, and room temperature were compared with the control storage condition (−80°C). In a follow-up study, the stability of storage of 14 samples was evaluated at 4°C at days 1‒3 and 7 compared with a control storage condition (−80°C) to establish the duration of time samples that can be stored at 4°C. Five freeze-thaw cycles, performed at −65°C or below, were also evaluated using four MS serum samples compared with fresh samples.

3.1 Analytical characterization and validation

Experiments were performed between July 2020 and July 2021. Fifty-one plates were run (40 and 11 plates using the first and second lots of manufactured kits, respectively).

Based on the analytical validation and characterization of individual biomarkers described below, the 18 out of 21 biomarkers that were included in the algorithm were determined to have acceptable analytical performance. GH and COL4A1 were excluded from the algorithm based on the analytical characterization studies described below. VCAN was not incorporated into the final algorithm due to biostatistical factors unrelated to analytical performance.

3.2 Incurred sample reanalysis

All individual biomarkers were determined to have a mean %CV <20% and met established acceptability criteria (Figure 1A). The Disease Activity score and the four Disease Pathway scores demonstrated reproducible results throughout the range of scores (Figures 1B□1F). For the Disease Activity score, the average SD across 48 samples was observed to be 0.3 score units, which is less than one interval (0.5) on the reportable scale, and as a result, met acceptability criteria. Additionally, incurred sample reanalysis showed robustness and equivalency of the assay between lots and laboratories, with the exception of COL4A1 (Table S3).

• Download figure
• Open in new tab

FIGURE 1.

Incurred sample reanalysis results for (A) Individual biomarkers and (B) Overall Disease Activity score, (C) Immunomodulation, (D) Neuroinflammation, (E) Myelin Biology, and (F) Neuroaxonal Integrity pathway scores in the MSDA Test.

3.3 Assay accuracy, precision, and sensitivity

Samples for the accuracy assessment were selected from an internal MS cohort (n=64) to target both high and low concentrations for the individual biomarkers relative to the MS population. Twenty mixed samples from four selected samples were analyzed for each biomarker. Minimum percent recovery for each biomarker ranged from 78% to 89%; the maximum percent recovery for each biomarker ranged from 99% to 124%. The median percent recovery ranged from 91% to 100% (Figure 2A). Additionally, the Disease Pathway and overall Disease Activity scores were calculated for both observed and expected concentrations of the various sample mixtures. The observed calculated scores correlated with the expected scores; R² ≥0.85 was established as the acceptability criteria (Figures 2B 2F).

• Download figure
• Open in new tab

FIGURE 2.

Accuracy of the MSDA Test to detect (A) Individual biomarkers and (B) Overall Disease Activity score, (C) Immunomodulation, (D) Neuroinflammation, (E) Myelin Biology, and (F) Neuroaxonal Integrity pathway scores.

Twelve replicates per serum pool were analyzed on a single plate for the intraassay precision assessment; ≤51 values per serum pool were analyzed across 51 plates spanning 2 lots of reagent kits. The intra- and inter-assay precision satisfied the criteria for meeting the precision parameter with most analytes passing the established criteria. Of note, COL4A1 was found to have inferior inter- and intra-assay precision that ranged from 7% to 47% and 15% to 59%, respectively. Based on these findings, COL4A1 was removed from consideration for inclusion in the N=1645) were analyzed during the assay algorithm. MS serum samples (N=1645) were analyzed during the assay development and validation process and used to establish the MS reference ranges for each analyte. Sensitivity analysis demonstrated that the LLOQ and ULOQ of each analyte met the sensitivity requirements established for the assay. The maximum percentage of samples requiring imputation at any LOQ was 1.8% (for NfL at LLOQ) (Table 1).

3.3 Assay interference

Most biomarker interactions with interferent combinations, such as common MS drugs, DMTs, and mAbs produced a median recovery that ranged from 80% to 120% (Figure 3). A lower percentage recovery was observed for two biomarkers, COL4A1 and CCL20, demonstrating a potential alteration in the presence of the sample for individual drugs. COL4A1 produced a low percent recovery for several drugs that ranged from 71% to 79%, which was likely an artifact of established assay imprecision (Figure 3 and Figure S2). For CCL20, cefoxitin spiked at 660 mg/dL resulted in a median percent recovery of 77% (Figure S2). Additional assay interferents are shown in Figures S2 (common drugs) and S3 (routine endogenous interferents and heterophilic antibodies).

• Download figure
• Open in new tab

FIGURE 3.

Assay interference for common MS drugs, DMTs, and the high concentration of universal mAb surrogates in the MSDA Test.

3.4 Diurnal variability

Diurnal variation was evaluated in eight patients over six time points (Figure S4). Mean and median percent differences for each biomarker and patient were observed to be within ±20%; mean and median %CV was found to be <30% for 19 of the 21 biomarkers (Table 2. Of note, there were some individual samples that were outside of the acceptable range (±30%; data not shown). In addition, mean and median diurnal variability ≥30% was observed for COL4A1, which may have been due to the imprecision of the assay to detect this biomarker. GH was also found to be more variable compared with the other biomarkers, which is not surprising, as GH has been previously reported to have a high degree of ultradian and diurnal variability.[28] For this reason, GH was removed from consideration for inclusion in the algorithm.

3.5 Sample stability

In the initial stability study, all biomarkers were stable for up to 1 day at room temperature and at 4°C, and for 28 days at −20°C. For those samples stored at room temperature, CXCL13, IL-12β, and TNFSF13B decreased beyond −20% at 3 days. During a follow-up study, all biomarkers were found to meet acceptability criteria when stored at 4°C, and consistent with the initial study as well as the control Table S4 In a study to examine the stability of samples after freeze-thaw, most biomarkers met acceptability criteria when compared with fresh sample. Of note, GFAP concentrations decreased beyond −20% for freeze-thaw cycles 4 and 5 (Table S5). Finally, score level analysis showed that test conditions were within 3 SDs (±1.5 score difference) from the control conditions during the initial study (Table S6) and at follow-up (Table S7). From these findings, we showed that biomarker levels were found to be most affected above certain thresholds (room temperature for 24 hours, 4°C for 7 days, −20°C for 28 days, and three freeze-thaws). These data can be used to establish allowable sample handling and storage conditions. Beyond these empirical estimations of protein stability, it is also important to note that statistically meaningful associations of biomarkers with multiple MS endpoints were observed using samples that had been stored at −80°C for extended periods of time. This suggests that the target epitopes for the proteins that were selected for inclusion in the custom assay panel and the final algorithm were sufficiently stable to derive clinically meaningful insights.