Overton in research: exploring trends in policy-cited papers

New research from scholars at Renmin University of China looks at how knowledge from scientific papers makes its way into government policies, to see if they could identify trends about the type of research that is cited in policy.

Here we’ll explore the paper and how the Overton data helped them draw their conclusions.

What is the paper about?

TITLE: Study on the Characteristics of Cross-Domain Knowledge Diffusion from Science to Policy: Evidence from Overton Data

AUTHORS: Chao Ren, Menghui Yang

This paper explores how knowledge ‘diffuses’ from science to policy, studying it according to its strength, breadth and speed. The research helps us understand how science and policy interact, and offers insights into how evidence-based policymaking could be improved. 

In recent years - especially since the Covid 19 pandemic - governments across the world have become increasingly reliant on academic research to inform their decisions. While this is a positive overall trend, as it indicates that policymakers are listening to expertise, we don’t know a huge amount about the type of research being used. 

In this paper, Ren and Yang set out to remedy this. They looked at all policy published in the year 2021 as a starting point, identifying which of these cited academic research. They then constructed a citation network, based on the citation relationships between all of these academic papers and policies. They then analysed these papers based on a set of indicators - “diffusion strength, diffusion breadth, and diffusion speed”* - to establish some characteristics of knowledge transfer from research to policy. 

*In this paper, diffusion strength refers to the amount of times a scientific paper is cited by policy documents. The more times it’s cited, the greater its impact on policy, including absolute and relative strength. Diffusion breadth means the different number of policy sources that cite a scientific paper, representing its coverage across different bodies. Diffusion speed relates to the rate at which a paper is cited in policy after its publication. 

The study had a number of general findings, including:

• The diffusion strength and breadth conform to a power-law distribution, while the speed follows a logarithmic normal distribution. This means that a small number of papers have the largest impact, getting cited frequently and widely. 
• Papers with the highest diffusion strength, breadth, and fastest diffusion speed generally come from world-renowned universities, scholars, and top journals.
• Papers in social sciences, especially economics, have high diffusion strength and breadth, while those in medical and life sciences have the fastest diffusion speed.
• The ‘Matthew Effect’ is apparent, meaning that successful well known academics or highly regarded institutions are more likely to have their ideas adopted in policies.

While many of these conclusions reflect general wisdom around science/policy engagement they did find variations in diffusion characteristics across different fields, as well as negative correlations between indicators, emphasising the complexity of the impact picture. For example, papers that spread quickly usually don't have a big reach or strong impact. This happens because fast-spreading papers often influence specific areas or topics of policy, instead of making a big impact across different subjects.

The researchers did also acknowledge potential biases in the data and note the impact of the COVID-19 pandemic on their findings.

How did they use the Overton data?

The researchers used the Overton database to collect metadata from all policy published in the year 2021. From the 588,786 policy documents found and analysised, the researchers identified a total of 1,164 government, international organisations, and think tank entities from 211 countries. 44,875 of these documents referenced scientific research, with 468,194 unique papers cited. They then extracted the policy data from these 44,875 documents before retrieving the metadata of the 468,194 cited scientific papers from the OpenAlex database, and using this to create a citation network.

Get access to the paper here to find out more about the characteristics of research impact in policy.

See a full list of papers using Overton data in our Zotero library.

Are you a bibliometric scholar interested in policy? Contact us and get free access to the Overton database to help you in your research.