Meta-Analytical Approaches to Ordinal Outcome Data in Clinical Interventional Studies: A Scoping Review with Reproducible Research

Scoping Review

Reproducible Research

Where sufficient data were available in the included studies or after contacting the authors, a reproducible analysis was conducted to replicate the original results. Statistical procedures were performed using RStudio (version 2022.12.0+353)⁷ and STATA 18 BE Basic Edition.⁸

Flow Diagram

Three hundred thirty-three records were identified: 245 from the database search and 88 from citation searches. Four studies were included in the review following a two-stage screening process (title/abstract and full-text). Out of the 108 studies assessed for eligibility, 104 were excluded. The reasons for exclusion were studies focusing on statistical methods for ordinal data in primary research (i.e., not within a meta-analysis context; n = 100), studies that reported meta-analytical methods for ordinal data in diagnostic accuracy studies (i.e., not within an interventional context; n = 3), and studies focusing on statistical methods to compute correlations between variables (i.e., not effect size for interventions; n = 1).

Figure 1 illustrates this review’s PRISMA 2020 flow diagram and the reasons for exclusion.⁹

• Download figure
• Open in new tab

Figure 1. PRISMA flow diagram of the Scoping review

Among the four included studies, one was available only as a conference paper, obtained after contacting the first and last authors. Despite its limited information, it was included because it contained relevant data.¹⁰

Characteristics of included methodological studies

All ordinal scales meta-analysed in the included studies had between 5 and 10 categories, with increments of either one or half points. So, the scales used in these studies had categories of 5 and 20. The number of meta-analysed studies (as samples) within each methodological study was relatively small, with counts of 5, 8, or 13. The exception was the conference paper, which analysed 216 studies across 24 meta-analyses but did not specify the number of studies in each analysis.¹⁰

Table 1 provides a summary of the characteristics of the included studies.

Methods to meta-analyse ordinal data

Three primary approaches for meta-analysis of ordinal data were identified. The first is the continuous approach, where the ordinal scale is treated as a continuous variable to calculate the effect size, using the standardised mean difference (SMD) as a metric. The second is the binary approach, where the ordinal scale is dichotomised at a specified cut-point to generate a binary variable, with the effect size calculated using the logarithmic odds ratio as a metric. The third is the ordinal approach, where the ordinal data is analysed in its original form. Within the ordinal approach, either a proportional odds model (frequentist or Bayesian) or a generalised odds model is used to calculate the effect size, with the logarithmic proportional odds ratio (logPOR) or the logarithmic generalised odds ratio (logGeOR) as the respective metrics.^11,12,13

For the binary and continuous approaches, the included studies in the meta-analysis reported the ordinal outcomes either as summary statistics (such as means and standard deviations or counts/proportions) or as individual patient data (IPD), which provides the distribution of subject responses across ordered categories for each treatment group. Both reporting styles were suitable for these approaches.

In contrast, the ordinal approach requires individual patient data to perform the analysis. However, the generalised odds model can also be conducted using median and interquartile range (IQR) as summary statistics.

Table 1 summarises the methods with their metrics and the reported data type.

Comparing the meta-analysis methods of ordinal data

The comparison of meta-analysis methods for ordinal data revealed that the direction of the effect size remains consistent regardless of the approach used.

The earliest methodological study in this area was conducted by Whitehead and Jones,¹¹ who utilised a proportional odds model to compare two techniques for calculating effect sizes: maximum likelihood estimates and Fisher’s information. This study is summarised in Table 1. A later study by Whitehead et al.¹² examined various models within the proportional odds framework, including comparing Bayesian and frequentist approaches.

The third study by Ashma Krishan and colleagues compared the binary approach with the proportional odds approach.¹⁰ However, this study was published as a conference paper with limited details. Despite contacting the first and last authors, no full-text paper was available for further analysis.

The final study by Cumming, Churilov, and Sena reported a comparison between the continuous method and the ordinal generalised approach (generalised odds model).¹³ This comparison was based on mock data generated from an ordinal outcome scale, with only one iteration of a meta-analysis (as described in Table 1).

Reproducible research analysis

One of the included studies,¹³ provided raw data, allowing us to conduct reproducible research. In this study, the authors used a neurological outcome scale, the Expanded Disability Status Scale (EDSS), to illustrate an ordinal approach (the rank-based generalised odds model) for meta-analysis of ordinal data and compared it with the continuous approach. The EDSS is used for multiple sclerosis and ranges from 0 (normal neurological state) to 10 (death), with half-point increments, except for the transition from 0 to 1, creating a total of 20 ordered categories.¹⁴

To generate the data for their analysis, the authors used the EDSS distribution from a large observational study¹⁵ to create a control group (sample 0) and five intervention groups (samples 1 to 5), each with 100 observations. They conducted a meta-analysis of five simulated studies, comparing each intervention group with the control group (e.g., sample 0 vs. sample 1, sample 0 vs. sample 2, etc.). The analysis was conducted using two approaches: calculating the standardised mean difference (SMD), representing the continuous approach, and calculating the log generalised odds ratio (logGeOR), representing the ordinal approach.

Their results showed statistically significant effect sizes in the same direction for both approaches (SMD = –0.17, 95% CI: –0.29 to –0.04; logGeOR = –0.21, 95% CI: –0.35 to –0.07). The heterogeneity was similar between the approaches (I² = 0% for both), as depicted in Figures 5 and 6 of the published report.¹³

We attempted to reproduce these results using the raw data provided in the supplementary material, conducting meta-analyses with both SMD and logGeOR in two software platforms: STATA and R. Our results differed from the original study, showing statistically insignificant effect sizes in both STATA and R (SMD in STATA = –0.03, 95% CI: –0.16 to 0.09; logGeOR in STATA = –0.04, 95% CI: –0.19 to 0.10; SMD in R = –0.03, 95% CI: –0.15 to 0.10; logGeOR in R = –0.04, 95% CI: –0.19 to 0.10). The heterogeneity remained consistent across both platforms and approaches (I² = 0%). Figures 2 (for forest plots in STATA) and 3 (for forest plots in R) illustrate these results.

• Download figure
• Open in new tab

Figure 2. Forest plots for our re-run of mocked-data in Cumming et al. 2015 using STATA, left using ordinal generalised odds model (with log GeOR as effect-size estimation) and right using continuous approach (with standardised mean difference as effect-size estimation)

• Download figure
• Open in new tab

Figure 3. Forest plots for our re-run of mocked-data in Cumming et al. 2015 using R, left using ordinal generalised odds model (with log GeOR as effect-size estimation) and right using continuous approach (with standardised mean difference as effect-size estimation)

To understand the discrepancies, we examined the raw data provided by the authors in the supplementary material, comparing the summary statistics with those reported in the original study. We found inconsistencies in three of the samples (see Figure 4).

• Download figure
• Open in new tab

Figure 4. Comparison of summary statistics of mocked data provided by Cumming et al. 2015, the upper table is from their report, and the lower table is through our calculations (the discrepancies are in red).

Despite these discrepancies, both analyses agreed on the direction of the effect size, and heterogeneity was similar between the two approaches. Further clarification from the original study’s authors would be helpful, but our attempts to contact them were unsuccessful.

Summarising the Literature

Despite more than thirty years since the foundation of the Cochrane Collaboration, meta-analysing ordinal data methods remain less well-established than binary and continuous data.^1,2 Our scoping review identified four methodological studies that addressed the meta-analysis of ordinal outcome scales with 5 to 20 ordered categories.

In 1994, Anne Whitehead and Nicola M. B. Jones proposed the stratified proportional odds model, combining multiple studies with differing classifications of ordinal outcomes. They used data from 13 trials examining the prevention of gastrointestinal damage with misoprostol, where the outcome scale ranged from 1 to 5, with one-point increments. This scale varied across studies, with some trials using only 2, 3, or 4 categories. They compared two ways of estimating logPOR and found no change in the direction of the effect size.¹¹

In a subsequent methodological study, Whitehead and colleagues introduced different proportional models to meta-analyse ordinal outcomes, including frequentist and Bayesian techniques.¹² They used individual patient data (IPD) for their analyses, allowing them to count the number of patients in each ordered category for each treatment group. The study included data from a subset of the same 13 trials from 1994 and another dataset from five trials evaluating the effect of tacrine in Alzheimer’s disease patients. The outcome scale in the second dataset was the Clinical Global Impression of Change (CGIC), a seven-point scale with one-point increments.¹⁷ This comparison also showed no change in the effect size between the two techniques of the proportional approach.

In 2013, Ashma Krishan and colleagues compared ordinal methods (using a proportional odds model) with binary methods (dichotomisation of the ordinal scale) using data from stroke trials in Cochrane systematic reviews.¹⁰ The outcome scale was the modified Rankin Scale (mRS), which has seven ordered categories from 0 to 6 with one-point increments.¹⁷ This study was available only as a conference paper, limiting its results’ depth, which indicates no change in the effect-size direction.

Finally, Cumming, Churilov, and Sena used the Expanded Disability Status Scale (EDSS) for multiple sclerosis, where EDSS ranges from 0 to 10 with half-point increments, to present the rank-based generalised odds model as an ordinal approach and compared it with the continuous approach. They also found no change in the direction of effect size between the two approaches.^13,14

Critical shortcomings in the existing literature

The existing methodological studies on the meta-analysis of ordinal outcomes have several shortcomings, limiting their capacity to provide robust empirical evidence for preferring one approach over another. The two studies by Anne Whitehead focused exclusively on different techniques and models within the proportional method, offering limited scope for broader comparisons. While the proportional odds method might be statistically appealing, in spite of its underlying assumptions,¹⁸ it requires effective implementation of raw data.^12,18,19 This reliance on individual patient data (IPD) makes it challenging to apply, as primary studies often do not provide sufficient data and there is currently no trend of complete sharing of raw data from clinical trials,²⁰ which leads to the exclusion of studies from meta-analysis.

On the other hand, the binary and continuous approaches can be performed using either IPD or summary statistics, offering more flexibility.² However, the binary approach entails dichotomization, which in turn leads to loss of information,²¹ and the continuous approach entails the assumption of normality of distribution, which does not always hold.^21,22 The study by Ashma Krishan, which compared the proportional odds approach with the binary method, was published as a conference paper with incomplete reporting, raising concerns about the validity of its findings. And the study by Cumming and others attempted a comparison between the continuous approach and the generalised ordinal approach (generalised odds model). However, this analysis was based on only one simulated meta-analysis with no iterations. Moreover, our attempts to reproduce their results yielded different outcomes, casting doubt on the reliability of their conclusions.

Strengths and limitations

In this review, we aimed to be comprehensive by conducting both electronic and citation searches, including screening the references of relevant studies. However, one limitation is that the screening and data extraction were performed solely by the main author due to the unavailability of other reviewers. Despite this, we believe this limitation did not significantly influence our findings, as the total number of citations screened was manageable for one reviewer.

A key strength of our review is our attempt to reproduce the results of previous studies, which is crucial for assessing the credibility of the evidence. However, we faced a limitation in this area as we were unable to contact the authors of the reproduced study to clarify the discrepancies in our results.