ABSTRACT
Background The treatment recommendation based on a Network Meta-analysis (NMA) is usually the single treatment with the highest Expected Value (EV) on an evaluative function. We explore approaches which recommend multiple treatments and which penalize uncertainty, making them suitable for risk-averse decision makers.
Methods We introduce Loss-adjusted EV (LaEV) and compare it to GRADE and three probability-based rankings. We define the properties of a valid ranking under uncertainty and other desirable properties of ranking systems. A two-stage process is proposed: the first selects treatments superior to the reference treatment; the second identifies those that are also within a Minimal Clinically Important Difference (MCID) of the best treatment. Decision rules and ranking systems are compared on stylized examples and 10 NMAs used in NICE Guidelines.
Results Only LaEV reliably delivers valid rankings under uncertainty and has all the desirable properties. In 10 NMAs comparing between 4 and 40 treatments, an EV decision maker would recommend 4-14 treatments, and LaEV 0-3 (median 2) fewer. GRADE rules give rise to anomalies, and, like the probability-based rankings, the number of treatments recommended depends on arbitrary probability cutoffs. Among treatments that are superior to the reference, GRADE privileges the more uncertain ones, and in 3/10 cases GRADE failed to recommend the treatment with the highest EV and LaEV.
Conclusions A two-stage approach based on MCID ensures that EV- and LaEV-based rules recommend a clinically appropriate number of treatments. For a risk-averse decision maker, LaEV is conservative, simple to implement, and has an independent theoretical foundation.
What is already known?A risk-neutral decision-maker should make treatment decisions based on Expected Value (EV), meaning that the single treatment with the highest expected efficacy from a network meta-analysis should be recommended, regardless of uncertainty. In practice, decision makers may recommend several treatments, and take uncertainty into account on an ad hoc basis.
What is new?We introduce Loss-adjusted EV (LaEV) as a mechanism for risk-averse decision making, and set out desirable properties of ranking systems. We define a ranking as valid under uncertainty if a higher EV is ranked above a lower one at the same uncertainty and a lower uncertainty above a higher one at the same EV. We compare LaEV to GRADE and probabilistic rankings. Of the methods examined, only LaEV provides a valid ranking under uncertainty and has all the desirable properties.
Implications For a risk-averse decision maker, LaEV is a reliable, conservative, and easy-to-implement decision metric, with an independent theoretical foundation. Adoption of a risk-averse stance might focus attention on more accurate quantification of uncertainty, and encourage generation of better quality evidence.
Competing Interest Statement
Howard Thom owns shares in the consulting company Clifton Insight which has received fees from Amicus, Argenx, Baxter, Bayer, Daiichi-Sankyo, Eisai, Kalvista, Merck, Novartis, Novo Nordisk, Pfizer, Roche, and UCB. No other authors have any conflicts of interest to declare.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
DATA AVAILABILITY
No new data were created or analysed in this study. The original NMA data and WinBUGs code are available from the cited guidelines.
