Who are COVID-19 guidelines & policy documents citing?

It's been great to see medical scientists embrace medRxiv as the repository of choice for new work on COVID-19. Pre-prints have never really taken off in medicine before but the pandemic has brought with it new ways of publishing and communicating research.

But what about its uptake? 

Usually policy documents don't cite preprints (outside of economics working papers) - has this changed too? Who're the journals, universities and think tanks currently feeding into government thinking and are there any surprises in the data?

Let's find out! At least at the 10,000 ft level - be aware that we're working with small numbers and citations are only one part of the picture.

If you're impatient then feel free to skip to the findings at the bottom of the page.

Finding COVID-19 related policy

Here's the Overton.io query I used to narrow down the search (looking only at documents published in 2020):

"2019-nCoV" OR "COVID-19" OR “SARS-CoV-2” OR "コロナウィルス"

OR  "코로나19" OR "코로나바이러스감염증-19" OR "مرض فيروس كورونا"

OR "coronavirus" OR "corona virus"

One limitation immediately jumped out, partly because of geographical biases in the index and partly because of language barriers: two thirds of the results are from the US and Europe.

We'll also have to accept that "coronavirus" will pick up some 2020 policy related to the flu in general rather than COVID-19 specifically.

In any case we get back 4,372 documents that together cite 4,632 different scholarly articles.

The 10 most commonly cited scholarly articles

Let's jump straight in and take a look at those cited articles.

The most commonly cited papers are all familiar: they're the headline articles you've probably already seen in The Lancet, JAMA and NEJM and are cited everywhere from Scotland to South Korea.

The Eurosurveillance paper is different: it's in the top ten because it's cited extensively by the European Centre for Disease Prevention and Control, the EU agency that's roughly the equivalent of the CDC in the US. Eurosurveillance is a journal it publishes.

Which scholarly journals have the most articles being cited

While those highly cited papers are all from high impact medical journals a different picture emerges when looking across the whole dataset. While The Lancet stays at the top now medRxiv and SSRN appear in 5th and 8th place respectively. Emerging Infectious Diseases, which is published by the CDC, sits at #4.

Here we're listing journals by the number of articles that have been cited at least once in the policy document set: if we order them by the number of citations instead then journals with broader "introductory" papers immediately jump up a few places:

If you've been focused on the medical science work going on you might be surprised that the American Economic Review and SSRN - the Social Science Research Network - appear in the top half of both lists.

That's because there's just as much thinking going on about the economic effects of COVID-19's accompanying fiscal crisis and the consequences of things like distancing, the long terms effects of lockdowns and human behaviour during pandemics (SSRN also publishes preprints from The Lancet, which gives them a boost).

Interestingly there are no big surprises in the journal list when you look at policy documents regionally, with one of two local exceptions (the Robert-Koch Institute's institutional repository makes an appearance when you look at Europe).

Which think tanks are being cited?

Scholarly work in journals makes up around 2/3rds of the citations in the COVID-19 policy document set. The other citations are to documents from government and third sector organizations like think tanks.

The Brookings Institution - the think tank cited the most often in the document set - has some good examples of the kinds of work being done by think tanks on the coronavirus. Its Strategies for coping with [..] the COVID-19 pandemic in Africa post is cited by an African Union report on the possible impact on African economies, and World Bank and Finnish government both cite a Brookings piece on the global macroeconomic effects of the outbreak in economic updates. Other articles outlining the places that COVID-19 recession will hit the hardest and the tools remaining to the Federal Reserve are cited in Congressional Research Service briefings aimed at US legislators.

While the top cited journal list doesn't change much from region to region the think tank list is different. Brookings, the American Action Forum and the Heritage Foundation are all cited primarily by North American sources in the COVID-19 policy documents set. Conversely The Institute for Fiscal Studies and the Elcano Royal Institute are both cited almost exclusively by European sources.

Which universities are being cited?

Last of all let's take a look at universities. To do that we'll use Overton's author affiliation data, which comes from the open (and excellent) Microsoft Academic Graph.

Across the Overton database as a whole policy sources have a tendency to cite work from local universities. This doesn't hold true for COVID-19 policy documents. 5 out of the 15 institutions with the most authors cited are Chinese:

Some of this is down to those popular articles we saw above - for example "How will country-based mitigation measures influence the course of the COVID-19 epidemic?" has authors from both Oxford and Imperial.

Wrapping up - what have we seen?

• The papers most commonly cited in COVID-19 policy are from high impact medical journals
• But medRxiv, SSRN and institutional repositories are also being cited frequently
• As are journals in economics in social sciences, as policy makers and think tanks look into the broader aspects of the pandemic
• Generally speaking policy documents favor local sources, but this doesn't hold true for COVID-19 related documents where Chinese universities make up 4 of the top 10 institutions cited in policy