ABSTRACT
Objectives Market access and reimbursement decisions for new Alzheimer’s disease (AD) treatments are informed by economic evaluations. An open-source model with intuitive structure for model cross-validation can support the transparency and credibility of such evaluations. We describe the new IPECAD open-source model framework (version 2) for the health-economic evaluation of early AD treatment and use it for cross-validation and addressing uncertainty.
Methods A cohort state transition model using a categorized composite domain (cognition and function) was developed by replicating an existing reference model and testing it for internal validity. Then, features of existing “ICER” and “AD-ACE” models assessing lecanemab treatment were implemented for model cross-validation. Additional uncertainty scenarios were performed on choice of efficacy outcome from trial, natural disease progression, treatment effect waning and stopping rules, and other methodological choices. The model is available open-source as R code, spreadsheet and web-based version via https://github.com/ronhandels/IPECAD.
Results In the IPECAD model incremental life years, QALY gains and cost savings were 21-31% smaller compared to the ICER model and 36-56% smaller compared to the AD-ACE model. IPECAD model results were particularly sensitive to assumptions on treatment effect waning and stopping rules and choice of efficacy outcome from trial.
Conclusions We demonstrated the ability of a new IPECAD opens-source model framework for researchers and decision-makers to cross-validate other (HTA submission) models and perform additional uncertainty analyses, setting an example for open science in AD decision modeling and supporting important reimbursement decisions.
INTRODUCTION
Market access and reimbursement decisions for new Alzheimer’s disease (AD) treatments are informed by economic evaluations. Recently, AD drugs lecanemab (1) and donanemab (2) have been tested in phase 3 randomized trials. They targeted persons with mild cognitive impairment (MCI) or mild dementia who had abnormal levels of amyloid pathology, a biological hallmark of AD hypothesized to cause dementia. The trial results showed significant reductions of amyloid pathology and slower decline on clinical scales measuring cognition and function. Other drugs are under development and being tested in phase 3 trials.
The European Medicines Agency is currently assessing the quality, safety and efficacy of lecanemab with a planned decision for market authorization early 2024 (3). The National Institute for Health and Care Excellence (NICE) in the United Kingdom (UK) is currently appraising the clinical and cost-effectiveness of lecanemab with a planned publication in July 2024 (4). NICE, based on a stakeholder workshop (5), identified key issues in the assessment of new drugs for treating AD. These included lacking evidence on the validity of surrogate endpoints, meaningfulness of the clinical outcomes and limited understanding of natural disease progression. Other issues are the potentially large budget impact, the role of new diagnostic pathways and risk of inequalities by disparities in access to treatment (6). NICE highlighted the importance of transparency and credibility of decision-analytic economic models, a known issue in policy making (7). The use of open-source models for healthcare decision making has been considered very important by most responders of a survey among stakeholders in academia, industry and HTA agency (8).
In their review of transparency in decision modeling, Sampson et al. (9) found that transparency is manifested through open-source modeling in addition to collaboration, peer review, reference models, reporting standards and model registration. In addition, the concept of model cross-validation has been defined as “examining different models that address the same problem and comparing their results” in good research practice guidelines (10) and has been argued it could increase confidence in models if similar results are observed.
In 2019 the IPECAD open-source model (version 1) has been developed (11). It has been re-used outside its developing team (12, 13), used for cross-comparing a single model (14, 15) and used for cross-comparison among multiple models (16, 17). Following recommendations, it reflected disease progression by multiple domains of cognition, function and behavior (18–20). However, we experienced difficulties implementing a treatment effect comparable to those observed for new AD treatments, requiring assumptions on dependencies between domains and calibration, which we think limited model transparency.
For the appraisal of current and possible future AD drugs, we argue that an open-source model with intuitive model structure that is easy to use for model cross-validation is urgently needed to support transparency and credibility of new AD drug cost-effectiveness assessments. Therefore, we describe the new IPECAD open-source model framework (version 2) for the health-economic evaluation of early AD treatment and aim to apply it in 3 use cases for AD lecanemab treatment: 1) cross-validating an existing model with a similar structure (ICER (21)), 2) cross-validating an existing model with a more complex structure (AD-ACE (22)) and 3) assessing additional uncertainty scenarios.
We intend the use cases to act as an example of how the new IPECAD open-source model framework could support the cross-validation of a model submitted for appraisal to a reimbursement agency and support addressing uncertainty. We note a detailed evaluation of lecanemab is outside the scope of this study.
METHODS
The new IPECAD open-source model framework was developed based on an existing cohort state-transition AD disease progression reference model assessing the potential health-economic impact of a hypothetical treatment in MCI due to AD (23). This model consists of states MCI and mild, moderate and severe dementia and death. We used original non-rounded transition probability input estimates available from the authors (AW and RH). The replicated model produced the same model outcomes in terms of mean person-years per state and alive when rounded to 2 decimal points, except for person-years in severe dementia after 40 years which had an absolute deviation of 0.01 person-years. We judged these outcomes as a sufficient reflection of internal validity. Next, we implemented new features for care setting, treatment stopping rules and treatment effect waning (i.e., decreasing of a treatment’s effect).
MODEL RATIONALE, DESCRIPTION, ANALYTICS AND ASSUMPTIONS
We opted for a simple, commonly used model structure with transitions between states and a 1-year cycle length to serve transparency and credibility to its users. Figure 1 shows the model structure detailing disease severity states as MCI and mild, moderate and severe dementia. It includes separate states for care setting and treatment status in MCI and mild dementia (i.e., on vs. off treatment) to allow various assumptions on treatment discontinuation and treatment effect waning.
Abbreviations: MCI, mild cognitive impairment; tx, on treatment.
The characteristics of the starting population include age, sex distribution and proportion in MCI and mild dementia (latter in off-treatment state for standard of care strategy and on-treatment state for intervention strategy).
Transitions among MCI and mild, moderate and severe dementia states reflect disease progression. Forward transitions (i.e., to more severe disease states) are allowed between all disease severity states. Backward transitions are allowed to all disease severity states except for transitions from moderate or severe dementia to MCI as these were not observed (24). Transitions are assumed to be time-independent (i.e., Markov assumption). This aligns with age not being a significant predictor (23), being a relatively small factor (25) or being adjusted for (24), but differs with progression dependent on time in state (26). Probabilities for remaining in the same state are calculated as 1 minus the probabilities for transitioning to other states.
Transitions from community to institution reflect changes in care setting. All disease severity states are duplicated for community and institution settings. The probability of transitioning from the community setting to the institution setting is conditional on disease severity, unidirectional (i.e., only from community to institution with no back-transition) and time independent.
Transitions to death can occur from any disease severity state. Mortality related to AD natural progression is assumed to be multiplicative to general population mortality. This is operationalized by multiplying the probabilities of death from a general population age- and sex-specific life table with the relative risk of death in each disease severity state. Due to age-specific mortality, the starting population reflects a single specific age and not an age range. Transitions between states other than death are assumed conditional on remaining alive during the cycle in question. We note transitions to death indirectly reflect time dependency in the form of age-specific mortality.
Transitions from on- to off-treatment reflect treatment discontinuation. Disease states for MCI and mild dementia in the community setting only are duplicated for on- and off- treatment. Disease states for MCI and mild dementia in the institution setting and for moderate and severe dementia in both settings reflect off-treatment. The probability from on- to off-treatment is independent of health state and unidirectional. This setup reflects the assumptions that treatment is discontinued at moderate dementia and that treatment is never provided in an institution care setting. Treatment discontinuation is operationalized as time-dependent probabilities of transitioning from on- to off-treatment. These are set over an initial period (e.g., due to side effects), a later period (e.g., due to health events interfering with receiving treatment) and a maximum duration.
Treatment effect relative to a control strategy is implemented as a relative risk multiplied to each of the transition probabilities from MCI to mild, moderate and severe dementia and from mild dementia to moderate and severe dementia. Specific relative risks can be set per transition between disease states and separately for on and off treatment. This latter feature facilitates consideration of a remaining treatment effect even after treatment is no longer provided.
Treatment effect waning is operationalized by a waning factor by which the treatment relative risk is raised (e.g., a treatment relative risk of 0.70 and 0.15 waning gives 0.70^((1-0.15)^cycle)). This can be specified separately for transitions on-treatment and, if applicable, transitions off-treatment.
Utilities (separate for patient and informal caregiver) and costs by health, social and informal care sector are specified per disease severity and care setting state (community or institution). Treatment costs are set to all on-treatment states. Caregiver utility was set to zero after patient death. No default setting is chosen and for this study aligns with the United States (US) setting for the cross-validation use cases.
A state trace (i.e., proportion of patients in each state at each timepoint) is calculated. A half-cycle correction is operationalized by taking the mean of each 2 adjacent timepoints for life years, QALYs and cost outcomes in each cycle. Cycle 1 is the period between time 0 and time 1, with starting age applied to cycle 1. The model allows for discounting life years, QALYs and costs. Inputs are provided per cycle to facilitate a different cycle length (detailed below). Multiplication between relative risks and transition probabilities are done on rate scale (27).
See supplemental material Table S1 column 1 and 2 for a list of all input parameters of the new IPECAD open-source model framework and a short description.
USE CASE 1: CROSS-VALIDATE EXISTING ICER MODEL
The first use case is the cross-validation of a model developed by the Institute for Clinical and Economic Review (ICER) in the US. ICER is an independent non-profit research organization that evaluates medical evidence and convenes public deliberative bodies to help stakeholders interpret and apply evidence to improve patient outcomes and control costs (icer.org). They developed a cohort state-transition model with AD disease severity states to estimate the lifetime cost effectiveness of lecanemab in comparison with the standard of care (21, 28). Its development is linked to an earlier assessment of aducanumab arguing the model conceptualization (29, 30) and its basic model structure dates back to another AD state transition model developed earlier (31). We sought to replicate all features of the ICER model given the similarities with the IPECAD model structure.
USE CASE 2: CROSS-VALIDATE EXISTING AD-ACE MODEL
The second use case is the cross-validation of a model published by Tahami Monfared et al. (22) which was funded by Eisai, the sponsor of the lecanemab phase 3 trial. They used the Alzheimer’s Disease ACE Simulator (AD-ACE) model (32) which uses a patient-level microsimulation approach capturing domains for AD pathology (e.g., beta-amyloid biomarkers), cognition, function, behavior and dependency. Its development is linked to earlier studies from Tahami Monfared et al. (33, 34), Kansal et al. (32) and Getsios et al. (35). We sought to mirror their input parameters for the cross-validation, given the different model type does not allow for replication.
USE CASE 3: ADDITIONAL UNCERTAINTY SCENARIOS
To show the potential of the new IPECAD open-source model framework we performed univariate sensitivity analyses on choice of efficacy outcome from trial, natural disease progression, treatment effect extrapolation (i.e., waning) and stopping rules, and other methodological choices. These factors were chosen as they have been shown to have (or are anticipated to have) a large impact on health-economic outcomes (5, 16, 17), are sometimes complex to implement (as compared to parameter uncertainty) or to our knowledge have not been extensively tested in earlier studies. These sensitivity analyses were performed using the lecanemab scenario as implemented by ICER. See Table 2 for details on these scenarios.
USING THE MODEL
The new IPECAD open-source model framework is available in multiple formats (R code, spreadsheet and web-based). The R version uses base R combined with the dampack package to facilitate model outcome visualization and sensitivity analysis. We followed a coding guideline (36) commenting the R code, using object prefixes and following a similar structure as R dampack vignettes (37, 38). The spreadsheet version (in open document spreadsheet format) and web-based (R shiny) version does not contain probabilistic sensitivity analysis functionality. The model is hosted on GitHub (https://github.com/ronhandels/IPECAD) for its features of version control and collaboration.
Most model features are optional and can for example be turned off by setting them to 0 (e.g., transition probabilities, utilities or costs) or 1 (e.g., treatment relative risks). Experienced users may add or adjust model features to suit their research objectives.
RESULTS
USE CASE 1: CROSS-VALIDATE EXISTING ICER MODEL
Almost all features of the ICER model were implemented. However, starting in institution care setting and age-specific utilities and costs could not be implemented and some features were open for interpretation. See supplemental material Table S1 and its table notes for a list of all input parameters and short description.
Compared to the ICER the IPECAD model mean per person outcomes showed higher life expectancy (+0.38 year), similar QALYs and lower costs (-$185,000) in the standard of care strategy. Incremental (intervention minus standard of care strategy) life years, QALY gains and cost savings were smaller (21-31% smaller) (see Table 1). The incremental cost-effectiveness ratio was somewhat larger ($272,000 instead of $236,000 per QALY gain).
USE CASE 2: CROSS-VALIDATE EXISTING AD-ACE MODEL
Where possible, features of the AD-ACE model were implemented except its structure and treatment effect waning due to its integration with structure. The reason was the fundamental difference in structure with AD-ACE a patient-level microsimulation on continuous disease progression domains cognition, function, behavior and biomarkers and IPECAD a cohort state transition on discrete disease progression composite domain of cognition and function. See supplemental material Table S1 and its table notes for a list of all input parameters and short description.
Compared to AD-ACE the IPECAD model outcomes showed higher life expectancy (+0.71 year), higher QALYs (+0.40) and higher costs (+$49,000) in the standard of care strategy. Incremental life years, QALY gains and cost savings were smaller (36-56% smaller) (see Table 1).
USE CASE 3: ADDITIONAL UNCERTAINTY SCENARIOS
Alternative scenarios were tested in univariate sensitivity analyses. First, the lecanemab trial primary efficacy endpoint of ‘relative difference in CDR-SB change from baseline’ and non-reported endpoint of ‘time shift in CDR-SB change from baseline’ was used as alternative to the ‘hazard ratio for progression to the next stage of dementia’. Second, an alternative source for natural progression from MCI to dementia was obtained from Vos et al. (39), and for natural progression between dementia states and death was obtained from Wimo et al. (23). Third, alternative extrapolating beyond the trial follow-up period were addressed in terms of ad-hoc combinations of treatment discontinuation (i.e., stopping rules) and treatment effect waning rates (see supplemental Figure S1). Fourth, an alternative method of a shorter cycle length of 1/24 month was employed.
Results of these additional uncertainty scenarios are presented in Table 2. As a result of extrapolation beyond evidence from the trial follow-up period more than half of the costs fell in the first 2 years while the majority of the benefits (QALY gain and care savings) were achieved after 2 years. Cost-effectiveness was lower (i.e., less net health benefit) in scenarios reflecting slower progression and treatment effect waning during treatment, and was higher (i.e., more net health benefit) in scenarios reflecting stopping treatment while assuming a treatment effect after stopping. We note the lack of evidence for stopping and waning assumptions related to the latter results.
DISCUSSION
We present a new IPECAD open-source model framework (version 2) for the health-economic evaluation of early AD treatment. We demonstrated its ability for cross-validation of other published analyses as well as additional uncertainty analyses. Health-economic outcomes seem related to model type and assumptions on treatment stopping rules and effect waning.
Differences with the reported ICER model outcomes could be explained by some features (e.g., start in institution care setting or age-specific utility and costs) not implemented in our IPECAD model. There may also have been ICER model features we interpreted or programmed differently. We note the SveDem model was closely replicated and post-hoc we relatively closely replicated another disease progression model by Herring et al. (14) (see supplemental Table S3). However, these two replications relied on support from the original developers. We think the limited replicability creates an opportunity for open-source modeling to improve transparency on model details.
Differences with the reported AD-ACE model outcomes could result from the assumption in AD-ACE that eventually patients would be in a similar state as if they had not been treated, which implies compression rather than postponing of the time spent in severe dementia. This would seemingly imply similar mortality between intervention and control, and thus fewer QALYs gained related to extending life. However, the AD-ACE model reported a larger QALY gain as compared to our IPECAD cross-validation. There may also have been AD-ACE model features we interpreted or programmed differently.
The additional uncertainty analyses showed that health-economic outcomes were sensitivity to assumptions on treatment stopping rules and treatment effect waning when extrapolating from the lecanemab trial 18-month follow-up period. We note calibrating the model to the time shift on the primary trial outcome required adjusting the model cycle length, which was also subject to uncertainty. In addition, the time shift scenario relied on the assumption that all aspects of the disease are shifted in time (i.e., a time shift in change from baseline on a continuous outcome translates to the same time shift in proportion in disease severity state).
The uncertainty scenarios confirm previous model comparison studies showing health-economic outcomes are sensitive to the choice of efficacy outcome from the trial and choice of natural disease history source (16, 17). Among recently reviewed studies (5) none addressed cycle time, and few addressed treatment discontinuation and treatment effect waning in scenario and sensitivity analyses. Whittington et al. (40) showed a lower incremental cost-effectiveness ratio with aducanumab treatment stopping at an earlier severity state and diminished outcomes with stronger treatment effect waning assumptions. AD-ACE model applications (22, 34, 41) showed higher lecanemab treatment value with higher discontinuation rates, with maintained reduced amyloid level after stopping treatment and with lower dosing frequency while assuming no treatment effect waning, and lower lecanemab treatment value with shorter maximum treatment duration. Kongpakwattana et al. (42) showed a higher cost-effectiveness ratio with earlier stopping donepezil at severe dementia. Ross et al. (15) showed a higher cost-effectiveness ratio with later discontinuation at severe instead of moderate dementia. An additional identified study (43) showed a lower ICER with a shorter maximum treatment duration, a lower ICER when assuming sustained effect after discontinuation due to amyloid clearance and a lower ICER when assuming treatment effects sustained longer. Another additional study (44) showed a higher ICER when assuming treatment effect waning. For the majority these results overlap to our finding of higher net benefit for higher discontinuation and optimistic treatment waning assumptions.
Recent cross-comparison studies in AD cross-validated models by comparing them after implementing a common benchmark scenario (16, 17). Although most previous modeling studies compared model outcomes to other studies (19), we identified only one example cross-validating a model by partly implementing another model’s scenario (14) in terms of starting age. We confirm the observation that differences between models are difficult to explain. Therefore, we advocate standardized reporting of undiscounted non-half-cycle-adjusted proportions in states over time to improve comparability of model outcomes to understand their differences (see supplemental Table S2) and sharing model outcomes using open science principles (for example on IPECAD repository https://osf.io/jv85a).
IMPLICATIONS FOR REIMBURSEMENT AGENCIES AND MODELERS
We think the new IPECAD open-source model framework can support transparency and credibility of new cost-effectiveness assessments for new early AD treatments. First, the model can be copied, adjusted or further developed with little effort and (staff or time) resources. Second, the model can cross-validate a model submitted by industry to increase the scope and rigor. Third, the model can address additional (uncertainty or subgroup) analyses, for example addressing uncertainties that have received little attention in previous research. We note addressing parameter uncertainty fell outside our scope but can be relatively easily implemented. Fourth, the model can be used for other (educational) purposes without any risk of confidential information being compromised.
In addition, our study results showed uncertainty related to assumptions on treatment stopping rules and effect waning. Because a detailed elaboration of these aspects falls outside the scope of our study, we recommend addressing in the assessment of lecanemab, donanemab and future AD treatments.
LIMITATIONS
Our study is subject to several limitations. We did not engage with patients and other stakeholders for the conceptualization of our model. However, another study has provided a rationale for the choice of a state-transition model type with similar disease states (30).
The model lacks reflecting diagnostic infrastructure to identify persons with abnormal amyloid eligible for treatment (45, 46), although this could be reflected by a number needed to test to identify a person eligible for treatment and corresponding test costs. Also, the model does not estimate the potential budget impact of new AD treatments, which likely is important to decision makers (6, 47). Also, the model does not reflect treatment effectiveness in specific subgroups such as APOE4, which could be associated to treatment effectiveness, adverse events and test costs.
The open-source nature of the model does not imply the validity of the model or any of its applications and does not imply it is error-free. Valid use implies adhering to scientific integrity standards for example in terms of transparency in selection of model inputs and assumptions (as compared to incorrect or ‘off-the-shelf’ application) (9). For example, careful consideration of implementing trial efficacy outcomes into the model as well as the choice for natural progression is important (17). Nevertheless, the open-source nature has the potential to reduce the presence of technical errors and facilitate incremental improvements (9), as well as to reduce the room for interpretation making it practically fully replicable.
Our model structure is of relatively simple nature, missing features for example to reflect treatment switching, domain-specific effects, alternative assumptions for mortality, efficiently address heterogeneity, treatment in institutional setting and time-dependent transitions between disease states. Also, we did not cross-validate the ICER state transition model with a microsimulation-type model. We note our previous IPECAD open-source model framework microsimulation version (version 1.2 available on www.ipecad.org) allows some of these features (48). Alternatively, the open-source nature of our model allows adding new features such as time-specific transitions (26) by skilled programmers.
CONCLUSION
A new IPECAD open-source model framework (version 2) was developed for the health-economic evaluation of early AD treatment. We demonstrated its ability for researchers and decision-makers to cross-validate other (HTA submission) models and perform additional uncertainty analyses, setting an example for open science in AD decision modeling and supporting important reimbursement decisions.
AUTHOR CONTRIBUTIONS
Concept and design: RH, AW
Development of the model: RH
Applications of the model: RH, WLH, SG, LJ
Analysis and interpretation of data: RH, WLH, SG
Drafting the manuscript: RH
Critical revision: ALL
FUNDING/SUPPORT
None
CONFLICT OF INTEREST
RH received outside this study research grants from JPND, ZonMW, IMI, H2020 (paid to institution); received outside this study consulting fees in the past 3 years from Lilly Nederland (2023), iMTA (2023), and Biogen (2021) (paid to institution); is member of IPECAD and member of ISPOR special interest group open-source models (un-paid).
WLH is a full-time employee of RTI Health Solutions, an independent nonprofit research organization. His compensation is unconnected to the studies on which he works. RTI Health Solutions did not receive funding for the current study. Outside of this study, RTI Health Solutions receives funding pursuant to research contracts with pharmaceutical and biotechnology companies. WLH is affiliated to research at the Karolinska Institutet and receives no compensation as part of this affiliation.
SG: has no conflict of interest to declare.
AW reports License holder of RUD-instrument (part) and Alzheimer Disease International MSAP scientific committee.
AS has no conflict of interest to declare.
LJ received outside of this study consulting fees from H. Lundbeck A/S, Novo Nordisk AS, Eli Lilly Inc and license fees for the Resource Utilization in Dementia (RUD) instrument.
BW received license fees for the Resource Utilization in Dementia (RUD) instrument.
SUPPLEMENTAL MATERIAL
Supplemental material related to manuscript entitled “New IPECAD open-source model framework for the health technology assessment of early Alzheimer’s disease treatment: development and use cases”.
A: Continue treatment & no treatment effect waning during treatment (base case)
B: Continue treatment & treatment effect waning during treatment
C: Stop treatment & treatment effect without effect waning after treatment stop
D: Stop treatment & treatment effect with effect waning after treatment stop
E: Stop treatment & no treatment effect after treatment stop
ACKNOWLEDGMENT
We wish to acknowledge master students Linh Nguyen (BSc) and Daphne Silvertand (BSc) for their support in the development of the model.
REFERENCES
Subject Area
Reviews and Context
0
Comment
0
TRIP Peer Reviews
0
Community Reviews
0
Automated Services
0
Blogs/Media
Author Videos