Silence is golden, by my measures still see: why cheap-but-noisy outcome measures can be more cost effective than gold standards

Calculation of the Variance From Categorisation for different distributions

In our analysis we assume that a study ‘outcome’ can be measured perfectly on an interval or ratio scale comprising 100 levels (0 to 99). For example, if the outcome was subjective wellbeing, then the gold standard would classify patients between 0 (lowest possible subjective wellbeing) and 99 (highest possible wellbeing) with each integer between indicating increasingly favourable outcomes. When no measurement of outcome is made at all, every patient is treated equally and is assigned to the same outcome, equal to the midpoint of the outcome scale. This is equivalent to using an outcome rating scale with just one outcome category. For our analysis we assume that this scale measures outcome as 49.5 (the midpoint of the range of the possible values 0 to 99) for every participant. We additionally model scales with 2, 3, 5, 8 and 10 categories, and assume that the outcomes are the midpoints of the ranges of possible values covered by each category. For example, the measured outcomes using a scale with two outcome categories are the midpoints of the ranges of possible values covered by each category: 24.75 (midpoint of the range 0 to 49), and 74.5 (midpoint of the range 50 to 99) (see Figure 1 and Supplementary Table 1 for more detail).

• Download figure
• Open in new tab

Figure 1:

Outcome rating scales that subdivide the entire range of outcomes into 2, 3, .. outcome categories.

We define the variance due to the difference between the true value of the observation and the value assigned to its scale category (when a continuous scale is categorised) as the Variance From Categorisation (σ_c²): where y_i,m denotes the true outcome value in participant i within category m an denotes the midpoint of the range of values in category m. M denotes the total number of categories and N the total number of participants.

We calculate the Variance from Categorisation for measures with 1, 2, 3, 5, 8, 10, or 15 outcome categories using a uniform distribution ranging from 0 to 99 and constrained discrete normal distributions with a range of 0 to 99, a mean of 49.5, and standard deviations ranging from 0.5 to 25 at 0.5 intervals. For the purpose of this study, we define no measurement as equivalent to having the population take a one category question which assigns everyone the mid-point value of the scale.

Analytic conditions for simple measures to be cost effective

The sample to achieve a defined power in a parallel study design with two groups is given by the sample size formula: , where n is the required sample size (assuming equal sized groups), F is a function of the critical values of the standard Normal distribution for a type I error of α and a type II error of β, σ₁ and σ₂ are the standard deviations in each group, and d is the difference in means to be detected. Using this formula and assuming that that there is no differential measurement error, we calculate the analytic conditions under which a simple measure will be more cost-effective than a more expensive but less noisy measure, where cost-effectiveness is defined as achieving the same power for a smaller cost.

Measuring the effect of participant burden

To estimate the effect of increasing participant burden on response rates we re-analysed the data from two systematic reviews identified in a Google Scholar search on 9/9/21, Edwards et al (2009), and Rolstand et al (2011).(7,14) We used the data provided in the systematic reviews as a source of sample sizes and odds of responding in each arm of the randomised control trials in which participants were randomly assigned to either a long or short questionnaire. Where the reviews provided information on the nature of the long and short questionnaire this was used to calculate the ratio between the two. Else, BW extracted this information from the relevant studies included in the review. Additionally, studies which Rolstand et al noted as non-randomised were excluded. We then regressed the logged ratio of the odds of responding on the logged ratio of the length of questionnaires using the metareg function in Stata (V16).(15,16)

Illustrative simulation of when simple measures are more cost-effective, and interactive web-app

Forming a definitive suggestion is difficult because many study specific factors will influence the equation on which design is more suitable. Such factors include the validity of the measures, the costs of both measures, the expected size of effect, and the amount of variability in the population. In addition, in practice, researchers generally do not measure measurement error using a variance statistic, but instead using correlation.

We therefore, additionally used a simulation to explore the comparative effects of using a cheap-but-noisy measure relative to a gold standard. In our simulation we model a power of 90%, an alpha of 5%, we additionally chose a low concurrent validity (r = 0.7) for the cheap-but-noisy measure, and a small effect size (mean difference = 0.1). Using parameters based on the CRASH-1 trial,(17) we model the cost of the gold standard as £50, and of the cheap-but-noisy measure as £5 per participant. We then measured the ratio of the costs of six measures (with 2, 3, 5, 8, 10, or 15 levels) for a range (0-25) of outcome standard deviations. In line with our previous simulations, we assumed the gold standard had 100 levels (0-99) and that that the outcome was normally distributed with a mean in the population of 49.5. Because the length of the gold standard was to some extent arbitrary, we ran the simulation with and without any effect of loss to follow up on response rates, additionally assuming that the cost of each lost participant was the same as the cost of a non-lost participant.

The simulations were produced in R,(18) and graphics were created using GG-Plot.(19) Because the parameters we chose may not be the ones of interest in practice, we also created an R-Shiny App, based off the above simulation, to allow researchers to use different parameters from the ones we have chosen.

Variance from Categorization for uniform and normal distributions

The variance from categorization for the distributions defined above, using scales with 2, 3, 5, 8 and 10 categories, are shown in Table 1 and Supplementary Table 2. We found that most of the Variance from Categorisation was removed after adding 3 to 5 response scale categories. For example, for the uniform distribution the scale with 2 outcome categories resulted in a 75% reduction in variance compared to a single category measure taking the mid-point value. Increasing the number of outcome categories used in the scale to 3 or 5 resulted in an 89% and 96% reduction respectively, when compared to a single category taking the mid-point value. The reductions in measurement error achieved by using scales with larger numbers of outcome categories therefore diminished rapidly (Figure 2). For the normal distributions, the σ_c²was also reduced by having a scale which was better calibrated to describing the variation in the data (demonstrated by the faster reduction in σ_c² as the standard deviation increased). The R code used to calculate the variance from categorisation can be found in Supplementary Table 3.

• Download figure
• Open in new tab

Figure 2:

Percentage reduction in variance from categorisation from adding measurement levels, compared to no measurement.

Analytic conditions under which simpler measures are more cost effective

The derivation of analytic conditions under which a simple measure is more cost effective than a noiseless but expensive measure, assuming equivalent measurement accuracy, can be found in Supplementary Table 4. This demonstrates that if the relative increase in cost from using the expensive measure is greater than the percentage increase in noise from using the simple measure, the simple measure will be more cost-effective.

The impact of participant burden on non-response

Edwards et al (2009) included 56 studies in their meta-analysis of the impact on questionnaire length on final odds of responding, and Rolstand et al (2011) included a further 8 randomised controlled trails not included in Edwards et al. Of these, two had insufficient information in either the review or the study’s report to ascertain either the ratio or actual length of questionnaire in each arm. Of the remainder, 51 measured the length of questionnaire using the number of pages, 8 used the number of questions, 2 provided word counts, and 1 provided the time needed to complete the questionnaire (Supplementary Table 5).

The linear meta-regression is presented in Figure 4 and explained 24.8% of the variance in the odds of participants responding. This showed that for every increase of 1 in the log ratio of the length of questionnaire the log odds ratio for responding increased by −0.570 (95% CI −0.851 to −0.289, SE = 0.140, p < 0.001). There was not, additionally, evidence that the intercept was different from zero (beta = 0.103, SE = 0.155, p = 0.509). We additionally did not find any evidence that adding in a quadratic term improved model fit (new r² = 23.42%). Converting these parameters from the log-log scale to the natural scale, we therefore find a power law relationship: the ratio in OR = RQ^-0.57, where OR is the ratio in the odds of responding to the long questionnaire relative to the short questionnaire, and RQ is the ratio in questionnaire length of long questionnaire to the short questionnaire. This implies that doubling the length of a questionnaire would reduce the response rate by around 65%, and therefore require asking around 50% more people to participate to achieve the same number of outcome observations (Figure 5).

• Download figure
• Open in new tab

Figure 3a:

Illustrative results of simulation, excluding the impact of non-response.

• Download figure
• Open in new tab

Figure 3b:

Illustrative results of simulation, including the impact of non-response.

• Download figure
• Open in new tab

Figure 4:

Results of the meta-regression, with both variables presented on the log-log scale. Both ratios are defined as the ratio in length of long questionnaire to short questionnaire.

• Download figure
• Open in new tab

Figure 5:

Illustration of the effect of increasing questionnaire length on reducing response rates from meta-regression. The y-axis can be interpreted as the multiple of how many more participants would be needed to achieve the same number of responses given how many times larger the questionnaire being used is.

Simulation and interactive web-app

Our simulation found that most cheap-but-noisy measures outperformed the gold standard when the standard deviation in the population was approximately greater than 1 (Figure 3). Ignoring the effect of participant burden means that the longer questionnaires, with a lower variance of categorisation, performed best (Figure 3a). However, including an effect for loss to follow up resulting in shorter questionnaires outperforming longer ones (Figure 3b). We additionally created an R-shiny app (available at https://benjiwoolf.shinyapps.io/cheapbutnoisymeasures/) to allow readers to further explore our simulation using different parameters. The R code used in the simulation is available in Supplementary Table 3.

Principle findings

We have shown that cheap-but-noisy measures, such as short questionnaires, may be more cost-effective than their gold standard counterparts. Specifically, a cheap-but-noisy measure will be more cost-effective than a gold standard measure with no measurement error or Variance from Categorisation if the relative increase in cost is greater than the relative increase in noise from measurement error and the Variance from Categorisation, providing the cheap-but-noisy measure maintains similar accuracy of the gold standard. This could be achieved by using well-validated short form questionnaires, and we would encourage further development and validation of such measures for a wide range of clinical outcomes.

We introduced the notion of Variance from Categorisation and showed that even a very simple measure, such as five yes/no questions or one five level Likert scale, eliminates the vast majority of the compared to functional no-measurement. was also reduced for normally distributed variables when the scale was calibrated to the variance expected within the population, with more homogeneous populations needing more sensitive scales. This demonstrates the importance of using a measure calibrated to the study population.

Our simulation demonstrates that the utility of a simple questionnaire, with few items or response categories, is inversely related to the required sensitivity of the test. If the measure is unable to detect effect sizes as large (or small) as they are expected to be, then the cheap measure is guaranteed to be less useful than a more sensitive measure, even if the gold standard is substantially more expensive. This is analogous to the final part of Smeden et al.’s Triple Whammy of Measurement error, that it can mask features of the data such as effect modification and non-linear associations.(20) Although this is an unsurprising finding, it raises an important caveat that the most appropriate measure will vary depending on factors unique to every study, and that our results do not warrant the use of cheap-but-noisy measures in every circumstance.

For many studies there is a cost associated with non-response. For example, when there is a sunk cost due to the allocation process/cost of the intervention, such as a pharmaceutical trial having to cover the cost of the active and placebo drugs taken by participant. Even if the intervention is free, an outcome measured through a postal questionnaire, e.g. in psychiatric trial, will have a cost associated with posting and printing unreturned questionnaires. An important implication of the association between participant burden (e.g. questionnaire length) and non-response is that, in studies with a cost of non-response, the analytic solution for when a simple measure is more cost-effect, will under-estimate the saving associated with using a simpler measure because the expected higher response rate. In a meta-regression of a previously conducted Cochrane systematic review we estimate that the ratio of response rates has an inverse power law relationship with questionnaire length, such that doubling the participant burden (measured as questionnaire size) will reduce the response rate by around one third.

One important source of systematic bias is differential response (selection bias). As this study shows, simpler questionnaires have meaningfully lower loss to follow-up. This implies that an additional advantage of the use of cheap-but-noisy measures is a reduction in the risk of selection bias. For example, the International Stroke Trial assessed disability after stroke for 19435 participants.[7] Conventional outcome measures (e.g. the Barthel Index and Oxford Handicap Scale) were considered to be too complicated and expensive. Instead, two simple questions with a reasonable validity relative to the Barthel Index and the Oxford Handicap Scale were used to measure handicap.[17,18] This allowed participants to be classified into three levels: needing help (‘dependent’), not needing help but still with some handicap (‘independent’), and those not needing help and with no handicap (‘independent and recovered’). At follow up, six months after randomisation, the trial achieved a 99% response rate, and evidence of a clinically important treatment effect. Even if all data was missing not at random, the potential for serious selection bias is minimal. It seems likely that this high response rate may have been influenced by the decision to use a simple outcome measure.

Strengths and limitations of the study

We did not consider two other types of non-differential measurement error: calibration error (which occurs when assigning incorrect units to a scale, for example using a yard stick to measure a metre without any unit transformations) and non-differential bias (which occurs when a constant number of units is added to all readings from a scale, e.g. parallax error).(21) Although both of these can invalidate results, psychometric-type questionnaires are typically standardised, and both of these errors are removed by standardising the outcome and calculating the standardised mean difference. Therefore, a caveat of our simulation is that we assume the scales are standardised.

The meta-regression should, additionally, be taken with a grain of salt. Firstly, neither review explored risk of bias, which is known to lead to an inflation of estimates.(3) The results may also not generalise to a setting relevant for our simulation. 82% of studies used the number of pages as their measure of questionnaire length, while our simulation explored the impact of varying the number of questions. The relationship between the number of questions and pages may not always be linear: two short questionnaires may both be one page but if the first is only half a page long the second may be almost twice as long while still being the same number of pages. Likewise, Edwards et al.’s review focused on postal questionnaires, and the point estimate may not generalise to electronic questionnaires. For example, we are aware of a more recent controlled trial exploring the effect of questionnaire length in electronic questionnaires which did not find evidence of a negative impact of longer questionnaires.(22) Finally, our meta-regression deviated from systematic review best-practices because additional information was extracted by only one author.

The parameters of the simulation obviously limit the generalisability of its results. Firstly, we only simulated a limited set of parameters which may not generalise. For example, it is unlikely that a large proportion of potential scales will have the same range and mean as the one we simulated. Because of this we created an R-shiny app (available at https://benjiwoolf.shinyapps.io/cheapbutnoisymeasures/) to allow researchers to use alternative parameters. Finally, it is worth noting that we assumed a certain level of validity in the cheap-but-noisy measures, and that our results do not warrant the use of completely unvalidated measures. We would therefore suggest that authors explore the use of validated short form questionnaires before constructing an unvalidated outcome measure.

Finally, that we assume a univariate analysis. Although this is appropriate for some study designs like randomised controlled trails and genome wide analysis studies, these results may not be applicable in other settings. Importantly, in a multivariate analysis, for example when adjusting for confounders or in a mediation analysis, non-differential measurement error can create a bias in either direction.(20)

Selection bias - Information Bias trade off

Our study has only explored the benefits of simple measures in terms of the power of a clinical trial. However, one important limitation is that simple measures may increase risk of (information) bias due to differential measurement error. Differential measurement error occurs when the error is not random but depends on some other factor related to the pertinent.(23,24) Risk of bias is arguably more important than improving power because random error can be eliminated in a systematic review and meta-analysis of many small studies, while residual bias is not. Simple measures, like a questionnaire, may be more suspectable to bias than a more expensive measure for two reasons.

Firstly, a simple questionnaire can be influenced by social and psychological factors which a more expensive objective measure, like a biometric reading, will not be. For example, a researcher may subtitle change the way they ask a question about alcohol consumption based of their perception of a participant (‘interviewer bias’), while a participant’s knowledge of social expectations may lead them to downplay how much alcohol they have drunk (‘social desirability bias’). On the other hand, neither the researcher’s nor participant’s expectations will influence a breathalyser reading. A well-designed questionnaire should be able to reduce this type of information bias. For example, psychometricians can use methods such as control questions, reverse coding, and an independent-rater to reduce the impact of these biases on study quality. A limitation of these methods is that they increase the complexity of outcome measures, and therefore may undermine both the simplicity and cheapness of the cheap-but-noisy measure. This type of bias can also be attenuated by improving the study design. Blinding can be used to reduce the possibility that any bias is differential across exposure status. Likewise, the use of anonymised postal or on-line questionnaires may reduce perceived social pressures or other (interviewer) bias due to having study personnel requesting information from participants in person.

A second source of differential measurement error could come from the measure itself. Many outcomes are intrinsically complex or multi-dimensional, obvious examples being socio-economic position or frailty. A risk of using a simple measure is that it may not capture all of the desired dimensions of the outcome of interest. When this occurs it is likely that the more simple measure will produce incorrect estimates of effect. For example, frailty is often thought to involve both physical and psychological dimensions.(25) An evaluation of an intervention designed to reduce frailty may produce misleading results if it only measures the psychological impact of frailty, or potentially miss the entire effect if the intervention’s impact is mostly mediated by reducing physical frailty.

Because these sources of information bias are intrinsic to the simplification process they are a limiting factor on the utility of cheap-but-simple measures. The higher participant burden in most gold-standard measures, however, increases risk of selection bias in studies that use them when compared to a cheap-but-noisy alternative. This implies the existence of a second quality-quantity trade off not explored in this simulation. The amount of selection and information bias in a study will vary depending on each study’s methods, and, if measurable, can only be quantified post-hoc. It is therefore impossible to provide universally applicable methods prescriptions beyond attempting to minimise the overall risk of bias in a study. With this in mind, we believe that cheap and noisy measures should be considered with caution in studies where information bias is likely, but may be useful in reducing the overall risk of bias in studies with a greater risk of selection bias than information bias.

Unanswered questions and future research

There are many outcomes which are continuous but may be easier to measure or model using a binary or categorical outcome. For example, recovery form a stroke is in theory a continuous outcome varying from no recovery (e.g. 0) to full recovery (e.g. 100). However, it has been argued that recovery can be efficiently measured using a four level ordinal outcome measured using two yes/no questions.(26–28) The practice of dichotomising continuous outcomes remains controversial because cut points are often arbitrary and generally reduce power.(29) One application of Variance From Categorisation may be as a conceptual aid in this discussion. It could be argued that, given justifiable thresholds, categorisation may reduce the total error when the Variance From Categorisation is less than the sum of the squared residuals from using a continuous parametric model for the outcome. As an extreme example, power law distributed variables with a negative exponent less than three, such as the supposedly common Pareto 80:20 distribution,(30,31) will not have a finite variance. Because they fail to meet a condition for the central limit theorem, the error in the estimation of their mean cannot be assumed to be normally distributed. Although beyond the scope of this paper, it is plausible that in such instances the total error in the statistical model for the outcome could be reduced by categorising it.

Finally, classical measurement error in the exposure of a bivariate analysis, such as a linear regression model with one regressor, will generally lead to regression dilution bias in which the effect estimate is biased toward the null. Various sensitivity analysis, such as regression calibration and SIMEX, have been developed to attenuate this bias.(32) To the extent to which a cheap-but-noisy measure contains more classical error, using these methods seems sensible. However, the extent to which these methods may be useful in overcoming any bias due to Variance from Categorisation remains unclear.

Conclusion and implications for outcome measures in clinical trials

Clinical trials will become more cost effective by employing cheap-but-noisy outcome measures, such as a simple questionnaire, when the relative increase in cost between the cheap-but-noisy measure and its alternative is greater than the relative increase in noise, assuming no bias. Simple questionnaires, with a given level of validity, have the added advantages of reducing loss to follow-up by improving response rates and not adding large amounts of noise. However, the relative merits of doing so will vary from study to study. Importantly, any increase in power and reduction of susceptibility to selection bias must be balanced against a potential increase in information bias. Box 1 provides a checklist of questions we hope will provide readers with a useful screen for when not to use a cheap-but-noisy measure. Finally, we have assumed throughout the existence of a previously created, and validated, questionnaire that could be used as a cheap-but-noisy outcome measure. Although questionnaires are becoming increasingly popular as health measures, see for example reference (33–36), we would encourage the development of a wider range of questionnaires to enable their use as end points in clinical trials.