RT Journal Article
SR Electronic
T1 Evaluating algorithmic fairness in the presence of clinical guidelines: the case of atherosclerotic cardiovascular disease risk estimation
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2021.11.08.21266076
DO 10.1101/2021.11.08.21266076
A1 Foryciarz, Agata
A1 Pfohl, Stephen R.
A1 Patel, Birju
A1 Shah, Nigam H.
YR 2021
UL http://medrxiv.org/content/early/2021/11/10/2021.11.08.21266076.abstract
AB The American College of Cardiology and the American Heart Association guidelines on primary prevention of atherosclerotic cardiovascular disease (ASCVD) recommend using 10-year ASCVD risk estimation models to initiate statin treatment. For guideline-concordant decision making, risk estimates need to be calibrated. However, existing models are often miscalibrated for race, ethnicity, and sex based subgroups. This study evaluates two algorithmic fairness approaches to adjust the risk estimators (group recalibration and equalized odds) for their compatibility with the assumptions underpinning the guidelines’ decision rules. Using an updated Pooled Cohorts dataset, we derive unconstrained, group-recalibrated, and equalized odds-constrained versions of the 10-year ASCVD risk estimators, and compare their calibration at guideline-concordant decision thresholds. We find that, compared to the unconstrained model, group-recalibration improves calibration at one of the relevant thresholds for each group, but exacerbates differences in false positive and false negative rates between groups. An equalized odds constraint, meant to equalize error rates across groups, does so by miscalibrating the model overall and at relevant decision thresholds. Hence, because of induced miscalibration, decisions guided by risk estimators learned with an equalized odds fairness constraint are not concordant with existing guidelines. Conversely, recalibrating the model separately for each group can increase guideline compatibility, while increasing inter-group differences in error rates. As such, comparisons of error rates across groups can be misleading when guidelines recommend treating at fixed decision thresholds. The illustrated tradeoffs between satisfying a fairness criterion and retaining guideline compatibility underscore the need to evaluate models in the context of downstream interventions.What is already known?Algorithmic fairness methods can be used to quantify and correct for differences in specific model performance metrics across groups, but the choice of an appropriate fairness metric is difficult.The Pooled Cohort Equations (PCEs), 10-year ASCVD risk prediction models used to guide statin treatment decisions in the United States, exhibit differences in calibration and discrimination across demographic groups, which can lead to inappropriate or misinformed treatment decisions for some groupsTwo theoretically incompatible fairness adjustments have been separately proposed for re-deriving the PCEsWhat does this paper add?Proposes a measure of local calibration of the PCEs at therapeutic thresholds as a method for probing guideline compatibilityQuantifies the effect of two proposed fairness methods for re-deriving the PCEs in terms of their impact on local calibrationIllustrates general principles that can be used to conduct contextually-relevant fairness evaluations of models used in clinical settings in the presence of clinical guidelinesCompeting Interest StatementThe authors have declared no competing interest.Funding StatementThe Stanford Medicine Program for AI in Healthcare and the National Heart, Lung, and Blood Institute grant 5R01-HL14455.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesBioLINCC - links listed below. https://biolincc.nhlbi.nih.gov/studies/cardia/ https://biolincc.nhlbi.nih.gov/studies/chs/ https://biolincc.nhlbi.nih.gov/studies/framoffspring/ https://biolincc.nhlbi.nih.gov/studies/jhs/ https://biolincc.nhlbi.nih.gov/studies/mesa/ https://biolincc.nhlbi.nih.gov/studies/aric/