Changes in medical scientific publication associated with the COVID‐19 pandemic

Rapid dissemination of information should not come at the expense of quality, ethical standards or oversight

The coronavirus disease 2019 (COVID‐19) pandemic has resulted in wide‐ranging health, social and economic impacts. By October 2020, global cases exceeded 41 million, with 1.1 million deaths.1 Urgent requirements for information were met with data on epidemiology, clinical features and recommended management being circulated on social media and pre‐publication servers. While this has allowed timely sharing of data, it has also brought risk of misinformation, with consequent changes to medical practice and misdirection of scarce resources based on flawed evidence.

Medical publishing uses peer review to provide independent and critical assessment to verify data integrity, validity of interpretations, and confidence in conclusions. This process can take many weeks; however, in a rapidly spreading pandemic, speed is a competing priority.

We hypothesised that these considerations may have altered the nature of medical publication. Accordingly, we characterised various aspects of COVID‐19‐related articles published in the five leading general medical journals with the highest impact factors (Web of Science) compared with an equivalent period in the preceding year.

Procedures for identifying, classifying and comparing publications were specified a priori. Research ethics approval was not required.

Publications were identified in the United States National Library of Medicine PubMed database. All articles published between 1 January and 31 May (inclusive) in 2019 and 2020 in The New England Journal of Medicine, The Lancet, JAMA, The BMJ and Annals of Internal Medicine were included. The sampling timeframe was defined by the first public health notification of COVID‐19 in China on 31 December 2019, ending at the time of the conduct of the literature search (Box 1). Within the 2019 search results, 60 articles were randomly selected using a random number generator in Stata 15.1. Publications without abstracts were excluded. Journal websites for each study period were searched for retracted articles.

Three reviewers independently abstracted the variables contained in Box 2 and Box 3. The h‐index (a measure of publication productivity and citation impact) of the first and last author was taken from Web of Science. A fourth investigator reviewed all data, harmonising interpretations and resolving any errors.

Data were analysed using Stata 15.1. Skewed continuous data were described using medians with interquartile ranges (IQRs) and compared using the Wilcoxon–Mann–Whitney test. Categorical data were compared using the Fisher exact test or χ² test as appropriate. Exact P values are reported and those less than 0.05 deemed significant.

During January to May 2020, PubMed listed 4001 articles, of which 1120 (28%) were related to COVID‐19. There were 134 articles with PubMed‐coded abstracts which were included for full review (Box 4). One additional COVID‐19 article was identified in the search for retracted articles but excluded from quantitative comparisons because it lacked an abstract. During the same period in 2019, 54 articles were ultimately identified as eligible for comparison (Box 4).

Compared with 2019, among the COVID‐19‐related publications in 2020, there were more case reports or case series, cohort studies, editorials and commentaries and fewer randomised controlled trials (7/134 [5.2%] v 19/54 [35.2%]) (Box 2). A similar proportion (37/52 [68.5%] non‐COVID‐19‐related articles v 74/134 [55.2%] COVID‐19‐related articles; P = 0.09) reported primary data. Of the 2019 articles, only two of 54 (3.7%) originated in China, whereas 32 of 134 (23.9%) of the COVID‐19 articles published in 2020 were from China.

The proportion of COVID‐19 articles in 2020 for which a correction was published was higher than for non‐COVID‐19 articles published in 2019 (28/124 [20.9%] v 4/54 [7.4%] respectively; P = 0.03). Time to the first publication of a correction was no different (median, 6 days [IQR, 4–14] v 7.5 days [IQR, 5–18] respectively; P = 0.53). Three 2020 COVID‐19 articles,2,3,4 but none of the 2019 articles, were retracted after publication. Only one journal, JAMA, routinely reported when a manuscript was submitted. In this journal, the median time from submission to publication fell from 139 days (IQR, 130–144) in 2019 to 23 days (IQR, 12–30) in 2020 (P < 0.001).

The median number of authors and their publication productivity and impact, as quantified by their median h‐indices, were similar. There was no statistically significant difference in the number of studies willing to share data under appropriate circumstances (P = 0.19), or those receiving commercial funding (P = 0.97).

The measured characteristics of randomised trials related to COVID‐19 were not statistically different to studies of an equivalent type published in the preceding year; however, numerically fewer subjects (median, 199 [IQR, 127–397] v 424 [IQR, 225–1076]; P = 0.07) and centres (median, 10 [IQR, 1–55] v 30 [IQR, 4–168; P = 0.15) participated (Box 3). Similarly, the observational study sample size was significantly smaller (median, 152.5 [IQR, 15–3481] v 191 972.5 [IQR, 1407.5–756 444]; P < 0.001), and the number of participating centres was numerically lower in the 2020 COVID‐19 group (median, 1 [IQR, 1–7] v 26 [IQR, 1–49]; P = 0.07). While not significantly different between groups due to the low numbers, 11 (16.7%) observational studies among the COVID‐19 publications did not report oversight by an ethics committee or institutional review board, and only nine (56.3%) case reports and case series with ten patients or fewer stated that patient consent had been obtained or that an exemption from this requirement had been granted.

In the first 5 months of the COVID‐19 pandemic, the five leading medical journals published a substantial number of articles that differed in many respects from their usual material. The journals examined were the clinically focused general medical journals with the top five Web of Science 2019 impact factors, ranging from 21.3 to 74.6, representing the medical literature with the greatest international influence on health policy and clinical practice. As reasonably expected, there was a seven‐fold reduction in the proportion of articles reporting randomised controlled trials, and a compensatory increase in small case series, opinions and editorials.

While there were few (n = 2) articles in the random selection of 2019 papers that were published from China, nearly one‐quarter of the COVID‐19 publications came from this country, as anticipated given the location of the earliest cases. There was no difference in the median h‐indices of authors, suggesting experienced academics pivoted rapidly to COVID‐19 research.

In circumstances which usually require consent, just under half of the COVID‐19 studies did not explicitly state consent was obtained, despite clear recommendations by the International Committee of Medical Journal Editors.5 The proportion of articles that referenced appropriate ethics committee or institutional review oversight was statistically unchanged; however, it is still a concern that 11 (16.7%) observational COVID‐19 studies lacked any statement to this effect. In addition, several other articles stated that they had been exempted from the requirement for ethical review due to the nature of the pandemic. Respect for personal autonomy and the value of independent oversight have always imposed additional workload on those seeking broader public health benefits. If COVID‐19 has created challenges in adhering to the usual practices of obtaining ethics approval and consent, consideration should be given to whether these processes could be amended to improve speed and accessibility, particularly during global health emergencies.

There was a near three‐fold increase in the proportion of studies that published corrections, perhaps reflecting the observed reduction in time from submission to publication observed in the one journal for which these data were available. It is likely this figure is an underestimation, given that corrections and retractions would be expected to continue over time. Three COVID‐19 studies were retracted. The publication of one of these articles4 had important implications, resulting in the temporary cessation of the World Health Organization's trial of hydroxychloroquine.6 While the corrections and retractions may be an artefact of increased speed to publication, it is also possible that their higher number might be the effect of enhanced focus on research related to COVID‐19. Nonetheless, journals must retain the integrity of review processes if they are to offer value beyond alternative online means of information dissemination.

This review has found similar results to bibliometric studies relating to the COVID‐19 pandemic, which have identified higher numbers of case series and reviews and fewer randomised clinical trials.7,8,9 We did not examine other articles from 2020 to understand the effect of COVID‐19 on contemporaneous publications, or to be able to comment on whether observed changes were specific to COVID‐19 or true of all 2020 articles. We note the convenience sampling of two similar periods may overestimate the magnitude of our findings. The cohort of 2019 studies for comparison was selected at random, rather than being matched by study type or size. When identifying h‐indices, we had difficulty identifying some Chinese authors, highlighting a bias against researchers without a name that can be distinctively rendered in the English language alphabet. Further implementation of unique author identifiers, such as the Open Research and Contributor ID (ORCID; www.orcid.org) or ResearcherID (Clarivate Analytics) would address this problem. We did not assess the quality of published studies or adherence to reporting guidelines.

As part of their early response to the worldwide problem presented by the COVID‐19 pandemic, there was a significant change in the characteristics of articles published by leading medical journals, with some evidence of a tendency towards publishing articles prematurely and those with lower internal validity. While these unique circumstances no doubt warranted such a change, rapid dissemination of information should not need to come at the expense of quality, ethical standards or oversight. Others have suggested several solutions to this challenge, including a two‐track review process for pandemic and non‐pandemic research, rapid preliminary assessment of research methodology by skilled in‐house reviewers before deciding whether to send for peer review, sharing of peer‐reviews between reviewers and journals, and mentored peer reviewing by research trainees.10 As part of pandemic preparedness, planning to facilitate augmentation of resources available to medical publishers, allowing maintenance of standards of review, should occur.

Box 4 – Publication identification flow diagram

---

COVID‐19 = coronavirus disease 2019.