Abstract
Epidemiologists have advocated for reporting confidence intervals and deemphasizing P values to address long-standing concerns about null-hypothesis statistical-significance testing, P hacking, and reproducibility. It is unknown if efforts to reduce reliance on P values have altered the distribution of P values. For 21,377 abstracts published 2000-2024 in 4 major epidemiology journals, two-sided P values (N=25,360) were calculated from estimates and confidence intervals scraped using ChatGPT’s 4o model. We evaluated trends over time to determine whether the empirical distribution of P values changed. We fitted to expected P value distributions and simulated these distributions with and without assuming changes in statistical power over time. Average P values decreased from 2000 to 2024, as did the fraction just below the 0.05 threshold. Fits to models indicate that statistical power increased. Increasing power would reduce average P value while also decreasing P values near the 0.05 threshold—precisely the trends observed in epidemiology journals. Although the frequency of P values near the 0.05 threshold has declined modestly, this likely reflects increases in statistical power rather than decreases in P hacking.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
NIH/NIA SFA R00AG073454 RC K00AG068431 MMG P01AG082653 JW F99AG083306 KDS K99AG083121 EF F31AG085965
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, University of Washington, and Vrije Universiteit Amsterdam.
Subject Area
Reviews and Context
0
Comment
0
TRIP Peer Reviews
0
Community Reviews
0
Automated Services
0
Blogs/Media
Author Videos