Abstract
Novel risk factors that improve statistical measures of fit on addition to established clinical prediction models often minimally change measures of discrimination (ROC curve AUC or c-index). As a result, measures of discrimination have been suggested to be insensitive in the evaluation of such models.
To understand this phenomenon, it is necessary to focus on the population risk distributions produced by models with and without the risk factor. This is because these risk distribution fully determines the risk distributions of cases/patients and controls/nonpatients, which in turn fully determine the ROC curve and its AUC. Broader population risk distributions result in larger ROC curve AUCs.
A fully independent risk factor with a relative risk of 2 added to the standard cardiovascular risk model produces risk distributions of those with or without the risk that are clearly different (which is evaluated by statistical measures of fit), while minimally broadening the population risk distributions (which is evaluated by measures of discrimination). The reason for this is that although addition of the risk factor replaces every risk stratum with higher and lower risk strata, this depopulated risk stratum is largely repopulated by similar splitting in neighboring risk strata. The interweaving of the the up and down migration paths to and from every point on the risk distribution results in a largely compensatory shuffling of risk assignments with minimal changes in the ROC curve AUC.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, University of Washington, and Vrije Universiteit Amsterdam.