Fake Publications in Biomedical Science: Red-flagging Method Indicates Mass Production

INTRODUCTION

Trust in the integrity of academic publishing is a foundation of science, and lack of it damages its reputation (Behl, 2021; Byrne, 2019; Seifert, 2021; Else & Van Norden, 2021; Else, 2022; Byrne et al., 2022). Well-known cases of scientific misconduct by individual researchers include ghost and “honorary” authorships (Flanagin et al., 1998; Wislar et al., 2011; Frederickson and Herzog, 2021), cherry-picking, abstract spin, plagiarism of images (Bik et al., 2016), and outright data fabrication (Bik, 2020; Byrne and Christopher, 2020; Park et al., 2022). While individual fraud has been recognized for centuries, the recent emergence of commercial production of fake publications is a new and unprecedented development (Flanagin et al, 1998; Wislar et al., 2011; Mavrogenis et al., 2018; Byrne, 2019; Byrne and Christopher, 2020; Else and Van Norden, 2021; Sabel and Seifert, 2021; Sabel, 2022; Chawla Singh, 2022; Candal-Pedreira et al., 2022). The major source of fake publications are 1,000+ “academic support” agencies – so-called “paper mills“ – located mainly in China, India, Russia, UK, and USA (Abalkina, 2021; Else, 2021; Pérez-Neri et al., 2022). Paper mills advertise writing and editing services via the internet and charge hefty fees to produce and publish fake articles in journals listed in the Science Citation Index (SCI) (Christopher, 2021; Else, 2022). Their services include manuscript production based on fabricated data, figures, tables, and text semi-automatically generated using artificial intelligence (AI). Manuscripts are subsequently edited by an army of scientifically trained professionals and ghostwriters.

Although their quality is relatively low (Cabanac and Labbé, 2021), fake publications nevertheless often pass peer review in established journals with low to medium impact factors (IF 1-6) (Seifert, 2021). Some governments, funding bodies, and academic publishers are on the alert (Cyranoski, 2018; Mallapaty, 2020; Else, 2022; Candal-Pedreira et al., 2022), yet many scientists, journal editors, and learned societies appear to be surprisingly unaware that such publications exist at all.

Paper mill customers – students and scientists – are pressured to publish in SCI publications by their academic or government institutions or university-affiliated hospitals (Pérez-Neri et al., 2022). For example, the Beijing municipal health authorities require a fixed number of first-authored SCI articles for physicians to qualify for promotion (Else and Van Norden, 2021). Academic policies that count publications and value impact factors as surrogates for scientific excellence can force graduate students to fulfill SCI publication requirements and pressure scientists and physicians to meet publication quotas to attain salary increases, promotion, and/or scientific reputation. Paper mills are ready to help and offer their services to accomplish these goals.

We are aware of several instances where paper mills tried to promote their business by inviting journal editors to collaborate, as shown by this unsolicited email in 2022 from a paper mill to one of us who is editor of a biomedical journal (see Tab. 1 for interview excerpts):

We are a well-known academic support institution from Guangzhou, China, which has been established for 8 years. … For reducing the publication time, we expect to cooperate with you in the future. Cooperation mode: we cite the content of your journal in our articles, thus increasing … your impact factor in 2022. You shall help us shorten the publication time. Payment: If an article is successfully published, we will pay for it at the price: IF*1,000 USD/article. For example, with IF=2.36, total payment=2.36*1,000 USD=2,360 USD. And this price is negotiable.

This attempted corruption motivated us to quantitatively analyze the global scope of fake publishing. Because the problem is still perceived to be small (an estimated 1 of 10,000 publications, Tab. 2), publishers and learned societies are just beginning to adjust editorial, peer-review, and publishing procedures. Yet the actual scale of fake publishing remains unknown, despite the fact that the number of reports on paper mills are increasing.

To be able to estimate the scope of fake publishing, a method is needed to identify potential (red-flag) fake publications (RFPs). We therefore looked for potential indicators of fakes that are easy to use by reviewers, editors, and publishers and tested their feasibility for randomly selected neuroscience and medical journals. We then developed classification (tallying) rules for screening for potential fakes and determined their sensitivity and false alarm rates. Here, we report the proportion of RFPs as an upper estimate of the true number of fake publications.

METHODS

Exploration

To search for potential fake indicators, in Study 1, one of us (editor of a neurology journal) sent a questionnaire to the corresponding authors of a sample of suspicious published articles and, for control, to those of a sample of unsuspicious articles (see Table 3). Based on the differential willingness to respond, we identified potential indicators for fake publications. These indicators should satisfy the following patterns (see below):

1. Authors of fake publications are reluctant to provide critical information as revealed by their response – or non-response – to the questionnaire by the editor,

2. the number of fake publications increases steadily over time, and

3. journals with a low to medium impact factor are most affected.

For Studies 1 to 6 we identified two easy-to-detect indicators, where a publication was labelled as RFP: if an author used a private email and had no international partner. We then tested these indicators in studies with 15,120 publications listed in PubMed® and Web of Science™ in the fields of neuroscience (including neurology) and (non-neuroscience) general medicine publications in an iterative way in a series of nine bibliographic studies (Fig. 1).

• Download figure
• Open in new tab

Figure 1. Screening plan for journals and publications.

Studies 1-6 identified potential indicators of fake publications on the basis of hypotheses 1 to 3 (see text), Studies 7-8 validated the indicators and developed a classification (tallying) rule, and Study 9 estimated the incidence of potential fake publications on the basis of the tallying rule from Study 7. Of note, some publications were used in several screening studies as follows: S2/S1 n=91; S4/S2 n=461; S3/S2+S4 n=707; S9/S4+S5=4001. The total number of unique publications analyzed is n=15,120.

Study 2 applied potential indicators to five neuroscience publications; Study 3 expanded this sample to estimate RFP growth from 2010-2020, including five journals in the field of general medicine. To estimate the 2020 incidence of RFPs, we increased the sample size in Studies 4 and 5. Finally, Study 6 checked the RFP rate in three open access journals.

RESULTS

EXPLORATION OF FAKE INDICATORS

We searched for indicators that can be determined easily, quickly, and reliably by an editor on the basis of a submitted manuscript or publication alone. In an exploration phase, our search was guided by three hypotheses:

Hypothesis 1: Authors of fake publications will be unwilling to answer quality check surveys and provide original data

In Study 1, n=215 neurology articles were manually inspected by an experienced editor; 20.5% (n=44) were deemed suspicious. A questionnaire was sent to all authors and, for control, to 48 authors of non-suspicious papers. It contained questions that authors of fake papers might be reluctant to answer (e.g., “Are you willing to provide original data?” [only 1 author of 44 suspicious articles did] and “Did you engage a professional agency to help write your paper? [none did]; see Tab. 3). Despite repeated reminders with a warning that failure to reply – or replying inadequately – could trigger retraction, the response rate among suspected authors was only 45.4% (20/44) compared with 95.8% (46/48) for the control group. This survey provided the first indicators of red-flagged fake publications (RFP).

Hypothesis 2: Because paper mills are on the rise (Else and Van Norden, 2021), indicators uncovered in Study 1, if valid, should also increase each year

Study 2 analyzed the frequency of these indicators in five randomly chosen neuroscience journals, expanded in Study 3 to a larger sample of articles from those five neuroscience journals and an additional five medical journals bi-annually (2010-2020). The results show a rapid growth of RFPs over time in neuroscience (13.4% to 33.7%) and a somewhat smaller and more recent increase in medicine (19.4% to 24%) (Fig. 2). A cause of the greater rise of neuroscience RFPs may be that fake experiments (biochemistry, in vitro and in vivo animal studies) in basic science are easier to generate because they do not require clinical trial ethics approval by regulatory authorities.

• Download figure
• Open in new tab

Figure 2. The rise of red-flagged fake publications (RFPs).

Number of publications and percentage of red-flagged fake publications in medicine (top) and neuroscience (bottom) (Study 3). The red line (y1-axis) shows the percentage of red-flagged publications per year; the grey bars (y2-axis) show the estimates of non-fakes and the red bars those of potential fake publications, extrapolated to overall articles published in neuroscience (selected from journals of the Neuroscience Peer Review Consortium) and medicine (selected from scimagojr.com). Note the relatively slow increase in potential fakes in medicine and the rapid rise in neuroscience.

Hypothesis 3: Because it is easier for paper mills to market their papers to journals with lower impact factors (IF; range 1-6) journals, our indicators, if valid, should also occur more frequently in such journals

Study 4 tested our indicators in an even larger sample of randomly selected journals included in the Neuroscience Peer Review Consortium. It red-flagged 366/3,500 (10.5%) potential fakes. In Study 5, RFPs were counted in 10 randomly chosen general (non-neuroscience) medical journals, where the 2020 rate was 23.8% in PubMed-listed journals with lower IF. Study 6 revealed that the RFP rate is even greater in open access (OA) journals (112/300 =40.3%) because it is often easier to publish there.

ESTIMATING THE INCIDENCE OF POTENTIAL FAKE PUBLICATIONS

The estimated incidence of RFPs in the Study 9 sample was 589/4,001, of which 328 (55.8%) were from China (Fig. 3; Tab. 4). The within-country percentages of RFPs vary considerably. Leading countries are Russia (48.3%), Turkey (47.5%), China (43.9%), Egypt (40.0%), and India (38.8%), with China – in absolute terms – as the largest contributor globally (55.8%). Note the large differences in fake rate between countries, even between neighboring countries with similar histories such as Russia (48.3%) and Ukraine (3.1%), probably due to different government policies (Fig. 3).

• Download figure
• Open in new tab

Figure 3. Estimating the global incidence of possible fakes in 2020

Left: Estimated percentage of red-flagged publications of each country’s publication output (based on Study 9). There are three clusters: countries with high (>30%), medium (10%-30%), or low (<10%) numbers of RFPs. Right: the contribution of different countries to the total number of potential fake publications (sample N=589). For absolute values, see Table 2. Interestingly, there are three clusters of the RFP rate which might reflect each nation’s average level of publication pressure: governmental (high), institutional (medium), or regular (low).

Given the 2020 global publication output of 1.33 million publications (Scimago) and an average of 28.8% RFPs in both fields of Study 3 (but 23.8% in Study 5), the 2020 RFP-incidence is approximately 383,000.

Assuming an average $10,000 price tag for a fake publication, the estimated annual revenue of paper mills is up to $3-4 billion. This revenue does not include non-Scimago journals (classified as “predatory journals”), which are probably more polluted by fakes, nor the open access and publication fees charged by academic publishers (approx. = $1 billion or an estimated 25% of paper mill revenue). Although the incidence of actual fake publications is expected to be smaller than that of RFPs if the number of false alarms exceeds that of missed actual fakes, the overall estimate of the annual number of potential fake publications is considerable (>300,000), and it is on the rise.

DISCUSSION

The dramatic rise of fake science publishing is driven by an unscrupulously corrupt – and increasingly successful – paper mill industry responsible for an estimated 380,000 RFPs annually as of 2020. This indicates that the number of actual fakes is likely higher than currently known (Tab. 2). A 2020 estimate of 28.8% of RFPs in the present study indicates that the 2011 estimate of 0.1% fake publications in China (Hu and Wu, 2013) and the 1% listed in Table 1 is too low, and is closer to the 21-32% “honorary” and ghost authorship cases in biomedicine (Flanagin et al., 1998; Wislar et al., 2011) and the 5%-10% reported in a pharmacology (Seifert, 2021) and a cancer journal (Heck et al., 2021). It feeds a billion-dollar global industry, magnitudes higher than the $4.5 million monetary value estimated in 2011 (Hu and Wu, 2013).

Fake science publishing is known to originate mainly from China (Hu and Wu, 2013; Lei and Zhang, 2018; Mallapaty, 2020; Schneider, 2021), India (Elango, 2021), and Russia (Abalkina, 2021), and, as we showed, it has evolved into a rapidly growing industry of fake science publishing. Our analysis confirms the existence, continuous growth, and notable scope of fake publishing, with most red-flagged publications coming from China (55.8% in 2020).

The rapid rise of the fake science industry is driven by SCI publication pressure on scientists, who are tempted to use paper mills that offer ghostwriting services at $1,000 to25,000 per publication, a profitable business model (Hu & Wu, 2013). Academic publishers acknowledge that the problem exists and are beginning to explore detection tools (Else, 2022; see also COPE & STM Committee on Publication Ethics, 2022). Chinese authorities, although aware of the situation (Cyranoski, 2018), have not yet resolved the problem. If quantity of scientific output is the index for becoming the world leader in science, then paper mills contribute to reaching this goal. In fact, China has almost caught up with the US in publication output (see: www.nature.com/nature-index/country-territory-research-output).

Paper mills feed on the rising administrative practice to evaluate researchers mainly by the “publish-or-perish” criteria of counting papers and journal impact factors as a surrogate for evaluating actual research quality and content (Van Dalen and Henkens, 2012; Candal-Pedreira et al., 2022).

Fake academic publishing is a major driver of global science publishing growth and a growing problem for medical practice. For example, Byrne showed that 712 problematic papers were cited >17,000 times and estimated that about one quarter of them may misinform future development of human therapies (Park et al., 2022). Preclinical studies at biotech company Amgen, for example, could replicate the results of only 6/53 “landmark” articles, and at Bayer, only 14/67 were replicable in oncology, women’s health, and cardiovascular medicine. The “replication crisis” slows down the development of life-saving therapies with an estimated financial loss of $28 billion annually by the pharmaceutical industry (Gigerenzer, 2018). Another example of how scientific fraud can affect medical practice is the report by Avenell et al. (2019). After assessing the citations of 12 retracted clinical trial reports in 68 systematic reviews, meta-analyses, guidelines, and clinical trials, they concluded that 13 out of the 68 reviews would likely have to change their conclusions if the retracted publications were removed.

It is important to keep in mind that our indicators provide a red flag, not legal proof, that a given manuscript or publication might be fake. However, it is the authors’ burden of proof to demonstrate that their science can be trusted. Whether this type of scientific misconduct is a conspiracy to commit injurious falsehood or a crime is for others to decide.

Fake science publishing is possibly the biggest science scam of all times, wasting financial resources, slowing down medical progress, and possibly endangering lives. The damage already done is unknown, and a realistic impact assessment of fake science is not yet available. The emergence of Chat-GPT and more sophisticated large language models might amplify the production of fake papers at less cost, although it will be difficult for paper mills to invalidate the indicators identified in this study (although deception via fake institutional email addresses also occurs).

Halting this development requires an immediate response. But what can be done? First, our simple detection tallying method can be used by reviewers and editors to red-flag potential fakes with or without additional indicators. Second, the academic community should consider revising its common practice to judge scientists’ productivity mostly (or solely) on surrogate quantitative criteria (publication numbers, H-factors, citation metrics, etc.) and instead evaluate the quality and relevance of their research (Van Dalen and Henkens, 2012). The European Research Council (ERC) has already taken a first step by asking researchers to refrain from listing impact factors in their applications, consistent with the San Francisco Declaration on Research Assessment (DORA). Thirdly, we need an advanced system to check scientific integrity, independent of academic publishers. Finally, learned societies, funding agencies, and governmental bodies should consider sanctioning fake polluted journals and their publishers.

Until science publishing fraud is largely eradicated, the collateral damage of fake science poses the risk that scientific analyses, experiments, and clinical trials will more likely fail, public health information will be less accurate or (intentionally) misleading, and presumably effective and safe therapies may not deliver what was promised. It also runs the risk that the public loses its trust in the honesty of science itself. Simple detection of fake publications, as proposed here, or more complex automated methods can help prevent further damage to the permanent scientific record and enable the retraction of fake publications at scale. We propose a “call to action” to restore the integrity of our global knowledge base in biomedicine, science, and technology.