Overton Blog
We were excited to see the ESRC’s new Transforming global evidence: AI-driven evidence synthesis for policymaking funding call from a couple of weeks ago… we’ve been thinking about this area a lot since growing our data science team last year.
There are lots of AI tools aimed at exploring and understanding scholarly literature - things published by academics in journals - but very few in the context of synthesising evidence for policy, where you need to draw on, qualify and assess a broader mix of scholarly, government and third sector inputs.
That data and those kinds of questions are our area of expertise! So we’re keen to join up and collaborate with others on a bid or related research projects.
We’ve picked out four examples of work that our data team’s doing that might be interesting to others in the space: if you’d like to learn more, have any ideas or are interested in working together then please do get in touch.
1. Using Overton Index to build an evidence base to inform Living Evidence Reviews
Overton Index is the world’s largest database of policy documents and grey literature. We track and index millions of full text documents in multiple languages from around the world, extracting bibliographic metadata, topics and references to the people and research they’re using as evidence.
This gives us an overview of the work being referenced in existing policy and makes Overton really useful for building a detailed map of evidence already being used in a given topic. Since the database is continuously expanding, finding updated content for these maps is quick and efficient.
Furthermore, many of the documents that we index are evaluations or reviews themselves. For example, searching Overton for policy documents with the terms “evidence”, “review”, or “synthesis” in the title yields tens of thousands of results that cite academic research.
2. Mapping the capacity to address government priorities at a national and regional level
We launched Overton Engage over the summer - it’s a large, searchable database of policy engagement opportunities. It allows users to see what national and regional governments around the world are seeking expert input on, and ties that to clear next steps for anybody who feels that they could contribute.
Our goal here is to make these opportunities more visible and lower the barriers to participation to ensure policymakers get access to the most relevant and targeted expertise, so there’s a basic version of this platform that’s free for researchers to use.
We also have a paid tier available for institutions, with extra features to help universities approach policy engagement more strategically, and build internal capacity. One of these features is ‘research matching’, in which suitable faculty are identified for each listing (and vice versa). We used an embeddings model to match scholarly abstracts to opportunities, then a ranking system to see which authors of relevant abstracts might be best aligned in terms of expertise.
This saves staff at institutions from having to pore over expert directories when finding relevant academics, but also means that we’re able to map - very approximately - where expertise across a set of universities exists for a topic, and where there may be a gap.
3. Mapping research to SDGs through the correct lens
Academic publishers are all in on the UN SDGs, to their credit. It’s common now for large publishers and bibliographic databases like Scopus and Dimensions to categorise research papers as relevant to one or more SDGs.
This is generally fine: you can easily tell that an academic paper about cleaning up ocean waste relates to SDG 14 (“Life below water”).
But the SDGs are not research goals, they’re policy goals. A better way of looking at things, we think, is to look at what research is being cited by policy targeting specific SDGs.
The IPCC reports cite lots of climate change research, of course, but they also cite social science and psychology journals. A paper on microfinance may not have any obvious keywords relating to gender equality, but it certainly relates to SDG 5 (“Gender equality”) if it’s used to create policy encouraging women in developing nations to start small businesses.
So, our SDG classifier helps our users understand the impact that they're having in these priority areas, by linking their research insights through to the policy that aims to address them. Our classifier uses machine learning to identify language related to specific goals, and categorises them accordingly. We built this classifier using an open-source training set from a group called OSDG - it contains thousands of text examples for all 17 of the SDGs, so we can train our system to correctly link documents to one of the goals.
4. Pulling it all together: exploring the evidence for policy ecosystem
We’re not an AI research company, but we’re pretty good at applying AI tools and techniques in our niche.
We’ve fine-tuned large language models to help link policy opportunities to relevant experts, grants and papers. Our models are specifically trained to bridge the gap between the language used by policymakers and the more technical, academic language found in journal articles and grant proposals.
This is an area we're passionate about, so we've done a lot of experimentation to explore potential workflows that leverage our data and AI capabilities. Lots of this work has been embedded into our existing products, or used as part of consultancy projects with external partners.
But if we were to bring it all together, what might this look like in practice? And how could it address the need for global evidence synthesis?
As a starting point we could proactively search Overton Engage for government priority areas to understand where there’s a need for evidence. By looking at these engagement opportunities, we can identify what governments across different regions are interested in or what their evidence gaps are.
Using an AI matching system we could then connect these interests to relevant projects or research. This can reveal who has the expertise needed to make well-informed decisions on these topics, so could be useful to policymakers who are looking to find partners or find the most current thinking on an issue.
Alternatively, we could surface policy documents that are relevant to a given opportunity using our custom embeddings model. This would be valuable for policymakers trying to understand the existing landscape, or for academics interested in discovering who has been publishing policy related to their discipline.
Here’s what this might look like for other governments and regions:
When it comes to synthesising the evidence, we could use AI to summarise the documents in a common format, and pull out relevant findings for each.
Finally, we could use our embeddings models to identify relevant experts linked to these topics, helping to fill evidence gaps and provide insights on the evidence syntheses we produce.
It could also be valuable to match researchers from different disciplines who share similar topic interests, encouraging collaboration between those who might not typically work together. This interdisciplinary approach would offer a more holistic way to address policy challenges.
These are all rough and ready methods using tools that, for the most part, are already available in Overton, and none of them are a direct substitute for expert human judgement - but hopefully it gives you a flavour of where our interests lie.
Interested in working together? Got other ideas that might be worth exploring? Get in touch! We’d love to collaborate.