ABSTRACT
Background Systemic racial and ethnic inequities continue to be perpetuated through scientific methodology and communication norms despite efforts by medical institutions. We characterized methodological practices regarding race and ethnicity in U.S. research published in leading medical journals.
Methods We systematically reviewed randomly selected articles from prominent medical journals: Annals of Internal Medicine, BMJ, JAMA, The Lancet, and NEJM within five periods: 1995-99, 2000-04, 2005-09, 2010-14, 2015-18. Original human-subjects research conducted in the U.S. was eligible for inclusion. We extracted information on definitions (conceptualization), measurement/coding (operationalization), use in analysis (utilization), and justifications. We reviewed 1050, including 242 (23%) in analyses.
Findings The proportion of U.S. medical research studies including race and/or ethnicity data increased between 1995 and 2018. However, no studies defined race or ethnicity. Studies rarely delineated between race and ethnicity, frequently opting for a combined “ethno-racial” construct. In addition, most studies did not state how race and/or ethnicity was measured. Common coding schemes included: “Black, other, White,” “Hispanic, Non-Hispanic,” and “Black, Hispanic, other, White.” Race and/or ethnicity was most often used as a control variable, descriptive covariate, or matching criteria. Under 30% of studies included a justification for their methodological choices regarding race and/or ethnicity.
Interpretation Despite regular efforts by medical journals to implement new policies around race and ethnicity in medical research, pertinent information around methodology was systematically absent from the majority of reviewed literature. This stymies critical disciplinary reflection and progress towards equitable practice.
Funding Funding was provided through training grants from the Eunice Kennedy Shriver National Institute of Child Health and Human Development [T32 HD091058] and the Department of Sociology, UNC Chapel Hill. Carolina Population Center provided general support [P2C HD050924, P30 AG066615]. NRS received additional support from the National Cancer Institute [T32 CA057711].
INTRODUCTION
Following global protests for racial equity, a growing number of health researchers are studying racism as a fundamental cause of morbidity and mortality. Such investment is long overdue. However, racism-focused work must be coupled with sound methodological practices surrounding the social constructs of race and ethnicity. Effective use of these constructs is integral to documenting and understanding how systems of racism and ethnocentrism affect health. Unfortunately, practices surrounding race and ethnicity in medical research are often absent of careful consideration, including methodological problems with definitions, measurement, coding, analysis, and interpretation of findings. The perpetuation of problematic practices maintains an ethnocentric status quo and may contribute to challenges in understanding how racism affects health, hindering effective and equitable healthcare and policy-making.
Debates over appropriate methodological decisions regarding race and ethnicity are longstanding. In the 1990s, researchers challenged the full range of methodological decisions: necessity of racial and/or ethnic data, construct definitions, choice of measurement, appropriateness of coding schemes, and role of variables in analyses.1,2 At the time, Thomas LaVeist (1996) argued that racial and ethnic data retained high utility for health research. He challenged health researchers to “do a better job” of conceptualizing race, understanding nuances of racial and ethnic measurements, and interpreting findings with care in order to help reduce health disparities in the United States (U.S.).3 Recent work in surgery and oncology have identified infrequent reporting of race and ethnicity data,4–6 however, no comprehensive systematic review of the state of these methodological practices in medicine over time currently exists.
The present study responds to LaVeist’s call and seeks to fill that gap by systematically reviewing trends in methodological practices regarding the conceptualization, operationalization, and utilization of race and ethnicity in U.S. medical literature. By examining publications in influential medical journals over the past quarter of a century, we document the state of medicine’s methodological norms and identify patterns of disciplinary practices that may reify misconceptions about race and ethnicity, with implications for scientific quality, reproducibility, and equity. In total, we investigated five core questions from a sample of U.S. medical publications: 1) What proportion of studies incorporate data on race and ethnicity? 2) What proportion provides conceptualization of race and ethnicity? 3) How is race and ethnicity data operationalized? 4) How is race and ethnicity data utilized in analyses? And 5) Do the authors justify their methodological decisions regarding race and ethnicity in the publication? We use this empirical evidence to inform suggestions for improvement at the level of authors, peer reviewers, and journals.
METHODS
Search strategy and selection criteria
The purpose of this study is to systematically review and characterize the methodological treatment of race and ethnicity in U.S. medical literature published in influential medical journals between 1995-2018. This study is a methodological systematic review under Munn et al.’s taxonomy,7 as the foundational methodological treatment (i.e., definitions, measurement, coding, analytical use, and scientific justifications) of two key variables - race and ethnicity - is the focus of this investigation. We define race as a social and political construct whereby social meanings (e.g., beliefs about ability, health, worth, etc.) are assigned to arbitrary phenotypes and which captures differential access to power, opportunities, and resources in a race-conscious society.8 Similarly, we define ethnicity as a social construct, stemming from a sense of belonging over shared cultural elements (e.g., language, religion, traditions, values) and/or of place (e.g., national origin).8 Both race and ethnicity are contextually, temporally, and geographically specific; neither race or ethnicity are biologically determined. For the purpose of this review, “Hispanic” and “Latino/a/x/e” are defined as a pan-ethnic identities, not as racial identities. Furthermore, “African American” is defined as an ethnic identity and is not synonymous with “Black.” See supplement 1 for background and rationale. Capitalization practices were not collected from sampled articles; however, we follow the AMA style guidelines for capitalization of racial and ethnic groups and capitalize all racial and/or ethnic terms in this article.9
The target articles under study include all U.S.-based, original, human subjects medical research published in Annals of Internal Medicine, BMJ, Journal of the American Medical Association (JAMA), The Lancet, and the New England Journal of Medicine (NEJM) between Jan 1, 1995 and Dec 31, 2018 (figure 1). Journals were selected based on impact factor and reputation, consistent with other methodological systematic reviews.10
Studies were identified by searching PubMed for empirical work published between Jan 1, 1995 and Dec 31, 2018. To reduce ineligible articles the following search terms were used: (English[Language]) NOT (Letter[Publication Type]) NOT (Comment[Publication Type]) NOT (Editorial[Publication Type]) NOT (Review[Publication Type]) NOT (News[Publication Type]) NOT (Case Reports[Publication Type]) AND ((“United States” [MeSH]) OR (“United States” [tw]) OR America[tw] OR “U.S.” [tw] OR “US” [tw]). Given the number of articles over the time period of interest returned by the original search (35,194; figure 1) and the richness of the data we aimed to collect, we took a simple random sample of 210 articles from five, five-year periods (1995-1999; 2000-2004; 2005-2009; 2010-2014; 2015-2019; 1050 articles total). Data collection occurred between July 2019 and November 2021.
All human-subjects research conducted exclusively in the U.S. was included. Non-U.S.-based research or multi-national research was excluded because of the unique social and geopolitical structures through which race and ethnicity function. We encourage researchers in other countries to conduct similar reviews using language and racial and/or ethnic categories that are important and specific to their context. Letters to the editor, commentaries, meta-analyses, and simulation studies were excluded. No restrictions were made on study outcome or exposure.
Data abstraction
Full details on the protocol have been reported elsewhere (unpublished data; under review). In brief, all included articles were independently reviewed in-full by two reviewers; data were abstracted into a standardized REDCap form.11 Abstraction was conducted using an existing protocol and all reviewers were primed using five to ten practice articles. Any abstraction discrepancies were discussed between the pair of reviewers, and if consensus could not be reached, were reviewed collectively by the entire author team. A third data quality check was conducted by the primary author. See supplement 2 for further details.
Software
Articles were sampled with Python 3.5.212 using Biopython and NumPy libraries. Analyses were performed in R, version 4.0.2.13 See supplement 3 for details.
Role of the funding source
Financial support was provided in part by training grants from the Eunice Kennedy Shriver National Institute of Child Health and Human Development [T32 HD091058] and the National Cancer Institute [T32 CA057711] with general support from the Carolina Population Center [P2C HD050924, P30 AG066615]. Additional pilot funding was provided by the Department of Sociology, University of North Carolina at Chapel Hill. Funding sources had no role in data collection, analysis, interpretation, or any aspect pertinent to the study.
Institutional Review Board
Study was found to be not human subjects research (NHSR) by the Institutional Review Boards (IRBs) of the University of North Carolina at Chapel Hill, North Carolina, United States.
RESULTS
From the 1050 screened articles, 242 were included (figure 1). The majority of excluded articles were either international studies or commentaries (figure 1).
Across time periods, the majority of studies were either cohort studies (range 56-73%) or randomized control trials (range 18-41%; table 1). Most studies examined a physical or mental health outcome (range 70-80%). “Other” outcomes were the second most prevalent (16-23%) and included studies on medical training, medical errors and the prevention of adverse events, or physician decision making.
Question 1: Inclusion of racial and ethnic data
The proportion of reviewed studies that included data on participants’ race increased over time (range 44-74%, figure 2). Studies that did not include participants’ racial data do not substantially differ from the overall sample with respect to study design, study outcome, or sample size (supplemental table 2). Over the same period, the proportion of reviewed studies that included participants’ ethnicity data has similarly increased (range 20-58%, figure 2).
Racial and ethnic data were almost always included together in the same study. Across all 149 studies which included participants’ race and/or ethnicity data, only a single study included data on participants’ ethnicity without also including data on participants’ race. When ethnicity data was included in the study, it was frequently combined with race into a single ethnoracial construct (range 81-100%). Only 11 (7.4%) studies across all strata included both race and ethnicity data and kept them as separate entities.
Question 2: Conceptualization of race and ethnicity
Across all 149 studies which included data on participants’ race and/or ethnicity, no studies provided a definition of either construct.
Question 3: Operationalization
In 59-90% of articles across strata, the measurement of race was “not stated or unclear” (table 2). In articles that indicated using “self-reported” race, it was frequently ambiguous if the measure was open-ended (i.e., free response) or close-ended (i.e., selection from preset options). Ambiguity between “open” and “closed” measures was more common in later strata (2005-09, 2010-14, 2015-18, table 2). Use of other measures (e.g., open, closed, and observed) was infrequent (table 2).
Results for ethnicity are similar; across all strata, articles commonly lacked any information on measurement of ethnicity (range 52-89%, table 2). Ambiguity between open and closed measures was more common in later strata (2005-09, 2010-14, 2015-18, table 2), and other measures (e.g., country of origin) were rare.
Coding schemes were collapsed across sampling strata and stratified based on the use of a strictly racial, ethnic, or ethnoracial construct. Racial and ethnoracial coding schemes were more heterogeneous, while ethnic coding schemes were more similar (table 3). Although “non-White, White” and “nonWhite, White” are functionally the same, we made no attempt to collapse coding schemes based on similarity due to concern about the subjectivity of those decisions. The most common racial coding schemes reflected predominantly a binary racial framing centering “Whiteness,” while almost all of the ethnic coding schemes centered on “Hispanic” or “Latino” binary coding. In the most common ethnoracial coding schemes (i.e., those representing >5% of the sample) “Hispanic” - an ethnic group - is compared to the racial categories of “White” and “Black.” Ethnic, racial, and ethnoracial codings all included “ns (not stated),” where no information was provided in the article about how participants’ racial and/or ethnicity data was re-coded for the study. Supplementary tables 3 and 4 contain the complete list of racial and ethnoracial coding schemes, respectively.
Question 4: Use in analyses
Race and ethnicity were predominantly classified as “not of interest” in analyses (i.e., used as a descriptive covariate, confounder, or matching criteria; range: 64-84%; supplemental table 5). Only four studies across stratum used race and/or ethnicity as an exclusion criterion, two of which restricted analysis to solely White participants. In 10-25% of studies across stratum, race and/or ethnicity were “of interest” (e.g., specific group comparisons, effect measure modification, or predictive variable).
Question 5: Justification
Approximately 30% of the 149 studies across strata which included participants’ racial and/or ethnic data provided a justification for at least one of their decisions surrounding race and/or ethnicity (e.g., the relevance of race and/or ethnicity to the study question, choice of measure, generation of coding scheme, and why an analytical approach or use of the variable was appropriate; data not shown). No studies provided justifications for the selection of a particular measure (e.g., selection of close-ended, self-report question over an open-ended, self-report question). Three studies referenced National Institutes of Health (NIH) or other institutional guidelines with respect to decisions making on measurement and coding. As in Castro et al. (2014), authors explained “race was assessed by participant self-report, using National Institutes of Health race/ethnicity reporting standards and categories” (p.2085-2086).14
INTERPRETATION
We aimed to systematically review methodological practices regarding the conceptualization, operationalization, and use of race and ethnicity in U.S. medical research published in prominent journals between 1995-2018. We found that information specific to race and ethnicity was routinely, if not systematically, absent from articles. While inclusion of racial and ethnic data has increased since 1995, no studies defined either construct and most did not describe how race and/or ethnicity was measured. In some cases, the coding schemes of racial and ethnic variables were even omitted entirely. Most studies across time periods did not provide scientific justification for their choices with respect to race and/or ethnicity.
Scientific rigor relies on replication and validation, which is rendered impossible if core methodological decisions are not clearly communicated. Core methodology includes information on definitions, measurement, and coding of variables, as well as scientific rationale. Absence of such information may also impact interpretation of findings or their translation into interventions, especially when it is unclear who is under study and why. Lack of basic information on methodology threatens our ability to conduct responsible and rigorous science.
Scientific and cultural racism
Journal word limits provide a potential structural explanation for lack of clarity regarding race and ethnicity as they force difficult decisions. Descriptions of methodological choices regarding race and ethnicity may compete with information on foundational literature, study design, exposure, outcome, results, or interpretations for inclusion.
The absence of information could also reflect a misguided belief in the presumed universality of race: that what race is and is not, the number of racial groups, boundaries between racial groups, and the “scientific relevance” of race to medical research are invariably understood. If conceptually race and ethnicity are universally understood across temporal, socio-cultural, and geopolitical contexts, then “race” does not need explanation or justification. Race, however, is not universal. Rather, what “race” is, the number of and boundaries between “racial groups,” and mechanisms by which the multilevel system of racism operates are deeply contextual. A large body of literature has theorized on how the social construction of racial and ethnic categories is historically situated and changes over time and place.8,15–18
The U.S., for example, is a nation explicitly designed to prioritize the life chances of a single group of people. As a settler-colonial state which achieved global financial power through slave labor and imperialism, the structures which continue to support the political, financial, judiciary, and educational systems maintain a hierarchical status quo based on established racial groups.19 Racism may be globally pervasive, but the structure of the system and the experience of living within it is different in the U.S. than it is in Mexico, Brazil, South Africa, India, or any other country.
Researchers cannot address health disparities in the U.S. without acknowledging the role that structural racism plays in health. In part, this requires naming the methodological assumptions behind the use of race and/or ethnicity in medical research. For over 150 years, medicine as a discipline was active in reifying the biological essentialism definition of race - that perceived behavioral and health differences between “racial groups” were true, immutable, and inherent to an individual’s genetic makeup. This pseudoscience has deeply infiltrated scientific institutions and thought despite the scientific process demonstrating the falsity of these claims. Scientific disciplines, however, spent so long justifying these ideas with “evidence” that race became “common sense” and perceived as part of the natural world.20 Within these structures, medical research in the U.S. has historically adhered to practices through which subordinated groups suffered as research subjects while resulting knowledge production benefited those of the dominant group.21 This practice contributed to the current state, in which racial and ethnic minorities are often systematically excluded in medicine, as both research participants and researchers.22,23 Thus, medical knowledge is predicated on only some bodies, cultures, and experiences. The lack of diverse perspectives contributes to the perpetuation of unconscious bias and racist practices in medicine.24,25
Institutions and structure
In light of ongoing conversations regarding the role of structural racism in U.S. health equity, journals and other institutions have developed communication guidelines around race and ethnicity. The International Committee of Medical Journal Editors (ICMJE) developed two such recommendations in 2004, namely that 1) the inclusion of racial and ethnic data is motivated and 2) the measurement of race and ethnicity is clearly explained.26
All of the journals sampled in our study aim to follow the standards set forth by ICMJE.27 However, for U.S.-based human subjects research published in these journals, adherence to these standards appears limited. After 2004, most studies still did not include information on how race and/or ethnicity was measured. Even considering the possibility of a lag between the release of new standards and the publishing of articles following those standards, adherence is low. Furthermore, few articles included justifications for decisions surrounding race and ethnicity.
Editors and specific medical journals have further echoed and elaborated upon the recommendations put forth by ICMJE. Following the ICMJE 2004 update, former JAMA Deputy Editor Dr. Margaret Winker introduced expanded recommendations specifically for network journals, calling for authors to provide details on (1) who assessed an individual’s race, (2) whether self-designation options were “open” or “closed,” (3) what the closed self-designation categories were, (4) if and how closed self-designation categories were combined, and (5) the rationale or relevance of race and ethnicity to a particular study.28 In supplemental analyses, there is minimal evidence of adherence to these additional higher standards among sampled JAMA articles (supplement 4). Recently, the AMA has released more explicit policies pertaining to definitions, capitalization, and the reporting of racial and ethnic measures, methods, and results.29
Actions for improvement
Previous work in medicine and adjacent disciplines has provided suggestions for methodological improvement.30 We build on this work by calling for clear communication of these improved practices in publication, including definitions, measurement, coding, use, and justifications. This is not a radical position. We simply argue that race and ethnicity should be given the same interrogation and justification as other variables, and that this be clearly communicated in publication.
The combined guidance around the communication of race and ethnicity offered by ICMJE, AMA, and other institutions is thoughtful and appropriate. Therefore we do not suggest new recommendations, we simply urge health researchers to follow existing guidelines. We similarly implore medical journals and editors to implement mechanisms for accountability to these standards. For example, authors could be prompted to certify at submission that they have adhered to ICMJE or AMA guidelines. At the peer-review level, additional training could be implemented to ensure that reviewers are confident in recognizing whether a manuscript meets criteria. We further encourage out of the box thinking to overcome structures; for example, pertinent details on race and ethnicity could be without word count, similar to human subjects statements or acknowledgements. Conducting annual reviews of policy adherence across medical journals could ensure that baseline benchmarks are being met.
Responsibility for meeting disciplinary standards of research falls on both medical journals and authors, as both are ultimately in service of patients and study participants. As “key players in the production of knowledge” (p.1288) and gatekeepers of research dissemination, editors and medical journals are in a unique position to ensure adherence to stringent scientific communication norms.31 In particular, prominent medical journals, by setting and requiring adherence to guidelines on clear communication, may influence disciplinary-wide standards. For authors, meeting these standards may require critical thought and conscious decoupling from earlier norms of conducting and reporting race and ethnicity in medical research.
Limitations
The abstraction from sampled articles is imperfect. The data retain a degree of subjectivity, despite protocols to standardize data entry and data quality checks. This is perhaps particularly true for the data on scientific justifications. Data abstractors were instructed to be as broad as possible when collecting information on justifications, thus data may be an overestimate of articles which included at least one justification. Second, it is possible that recent attention to addressing racism and ethnocentrism broadly has resulted in a renewed effort to “do a better job.” Subsequently, methodological practices and the communication thereof may have substantially shifted between Jan 1, 2019 and today. Finally, we did not review supplementary materials. If information on definitions, measurement, coding, or scientific justifications was included in supplements, they were missed.
Conclusion
Interventions aimed at addressing racism as a fundamental cause of disease in the U.S. must be based on unassailable research achieved through strict methodological rigor. Quality science enables knowledge democracy and health equity by providing a strong evidence base for changes in medical practice and policy. Dismantling systematic oppression in medicine requires clear, critical, and honest communication around the use of race and ethnicity data in medicine. This means that both the guidelines regarding the inclusion of race and ethnicity in medical research should be strengthened and editorial mechanisms should be created to ensure adherence. Collectively, the health research community needs to hold each other accountable to continue improving how race and ethnicity are conceptualized, operationalized, and utilized in medical research. This should be one element in a holistic, multipronged approach to addressing racism and health inequity which also centers additional systems reforms.
PANEL: Research in Context
Evidence before this study
Systemic racial and ethnic inequities can be perpetuated through scientific methodology. Effective use of race and ethnicity is integral to documenting and understanding how systems of racism and ethnocentrism affect health. Our review drew articles from five prominent medical journals selected based on impact factor and reputation (Annals of Internal Medicine, BMJ, JAMA, NEJM, The Lancet), indexed in PubMed, and published between 1995 and 2018. We selected a stratified random sample of articles from within 5-year increments (1995-1999; 2000-2004; 2005-2009; 2010-2014; 2015-2018) and randomly selected 210 articles per stratum for a total of 1050 articles. Inclusion criteria included U.S.-based, original, human subjects medical research. Twenty-three percent (242/1050) of sampled articles met inclusion criteria.
Added value of this study
This review provides a timely examination of race and ethnicity in medical literature. Our findings indicate that while the use of race and ethnicity in medical research has increased over time, information on definitions, measurement, coding, and scientific rationale was overwhelmingly absent. No studies clearly defined race and ethnicity. Information on measurement was frequently lacking. Race and ethnicity were typically not the focal variables of research, and few methodological decisions were clearly justified. We further contextualize our findings with respect to existing disciplinary guidelines (e.g., the International Committee of Medical Journal Editors “Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals”) surrounding race and ethnicity in medical journals. The majority of sampled publications do not appear in compliance.
Implications of all the available evidence
Inadequate scientific communication hindered our ability to analyze decisions made regarding the use of race and ethnicity constructs. Inability to identify key methodological decisions regarding integral variables in published medical literature prevents reproducibility and indicates systematic poor scientific practice. Systemic racism is a recognized determinant of health inequity within the U.S. and regular methodological shortcomings in the communication of medical research around race and ethnicity introduce barriers to addressing those inequities. Publishers have sought to incentivize more stringent and consistent guidelines around the use of race and ethnicity in medical research through the use of disciplinary guidelines like those of the ICMJE. However, based on our findings, U.S.-based research in prominent medical journals have largely fallen short of meeting existing guidelines.
As subjective individuals attempting objective science, our internalized biases, misconceptions, and personal beliefs leak into our science to influence who we include, why we include them, how we determine what is or what is not important to populations, how we interpret results, and ultimately what kind of interventions (and interventions for whom) we suggest. Adherence to guidance provides authors and the peer reviewers an opportunity to interrogate methodological treatment of race and ethnicity (why we did or did not measure it, why we measured it in a particular way, why we grouped certain individuals together, or centered certain individuals in the research but not others). In turn, we can possibly begin to identify how subjectivity impacts our research. If such information is entirely absent from publication, then we cannot begin to identify or address internalized misconceptions, let alone gaps in knowledge, treatment, and access related to racialized systems.
Data Availability
REDCap data entry form and full list of articles will be made available upon request with publication. Please contact the corresponding author for data inquiries.
CONTRIBUTORS
RAMM conceived of the study and directed its implementation, including quality assurance and control. All authors contributed to study design and data acquisition. Authors RAMM, NRS, and PNZ conducted analyses. RAMM and REW wrote the primary draft; all other authors (NA, ANG, NRS, PNZ) contributed to further drafts and edits. All authors had full access to and verified the data.
DECLARATION OF INTERESTS
We declare no competing interests.
DATA SHARING
REDCap data entry form and full list of articles will be made available upon request with publication. Please contact the corresponding author for data inquiries.
ACKNOWLEDGEMENTS
We are thankful to Dr. Allison E. Aiello and Dr. Robert A. Hummer for their guidance and support. We are indebted to Denise Mitchell for their assistance in data collection. Natalie R. Smith contributed to this work while at the University of North Carolina and is now a postdoctoral fellow in the Harvard TH Chan School of Public Health Department of Social and Behavioral Sciences.