Overton data in research: new paper on Covid education policy

New experimental research from academics at UCL explores how academic scholarship influenced government educational policy during the Covid pandemic, evaluating to what extent policy makers used the ‘best’ available research. 

Here we’ll take a look at this working paper, exploring the findings. We also chat with Basil Mahfouz about how and why they used Overton. 

What is the paper about?

TITLE: Tracking the impact of research on education policy during the COVID-19 pandemic

AUTHORS: Basil Mahfouz, Geoff Mulgan and Licia Capra

This paper analyses the scholarly citations of educational policy during the Covid-19 pandemic, to understand how governments used research during the pandemic. They were interested in the relationship between research considered as ‘excellent’ (by conventional academic indicators) and policy impact. Their ultimate goal with this preliminary case study was to tackle a new area where very little quantitative research has been done and potentially to start developing indicators to measure the abilities of policymakers to use academic evidence effectively.

Mahfouz et al wanted to understand whether or not policy makers were more likely to use the ‘best’ research - defined here as work authored by ‘top researchers, published in the most cited journals, conducted by the highest-ranking universities and with the highest numbers of academic citations’.

A conservative estimate suggests that there were more than 450,000 papers published on Covid in two years. Given this huge amount of data, the researchers wanted to understand how well policy makers leveraged the huge amount of research available to them.

Their initial findings suggested that policymakers tended to draw their research from a fairly narrow range of publications, generally relying on studies from their own countries. Significantly they did not find an observable link between ‘research excellence’ as defined by conventional metrics and the likelihood (and speed) of a paper to be cited in policy, suggesting that non-academic factors are more likely to influence policy makers and affect whether they incorporate research into their decision-making. 

They also found that policymakers tended to use medical research more effectively than educational research and that many of the cited publications were not strictly related to the topics of ‘school closure’ or ‘hybrid learning’ during the pandemic .The authors also acknowledged limitations in their methodological approaches and that further work would need be carried out to validate their preliminary results.

Using Overton to unearth policy

We spoke with lead researcher Basil Mahfouz about their use of the Overton database.

How did you use the Overton data?

“We started by identifying all research papers related to Covid-19 and education policy in Elsevier's International Centre for the Study of Research. As Elsevier is linked to Overton, we were able to see directly which papers were cited in policy using the Overton data. We were then able to analyse the ‘quality’ of these cited papers.

Interestingly, when we inspected the papers, I noticed a discrepancy - that the policies citing the research had nothing to do with Education - which got me to rethink the entire methodology.

So, the next phase of the study will involve reverse engineering the methodology - starting by identifying Covid-19 educational policies using our app and then tracking the underpinning research.”

Why did you choose Overton?

"Overton is the largest policy database in the world. With over 8 million policies indexed, I believed it was the most reliable option to identify the education policies published during COVID-19. Overton is already linked with SCOPUS data and can be accessed through the Elsevier International Centre for the Study of Research. The integration of these databases and the seamless transition across both allows for novel research questions to be tackled, at scale."

What did you like about the app?

“The Overton database includes the date of publication of the policy document, which can be compared to the date of publication of the document being cited. This enables us to explore a brand new metric: the speed of citation. Measuring this is useful because it helps us compare how quickly policy makers were able to incorporate the latest research across countries and disciplines. It also helps us understand what other factors influence whether or not a paper gets cited in policy.”

Was there anything you wanted to do with it but couldn’t?

“Unlike scholarly papers, policy documents don’t usually have an abstract or other standardised short description of the policy. I wanted to perform natural language processing on the policy documents as part of this analysis. Currently Overton provides a snippet for each citation, the sentences before and after an academic citation, but not a summary of the policy document. It would be great to have some standardised text summary that describes the content of the policies.”

See a full list of papers using the Overton data in our Zotero library.

Are you a bibliometric scholar interested in policy? Contact us and get free access to the Overton database to help you in your research.