Elsevier's latest survey shows desire for change to research impact assessment systems

Elsevier's new report ‘Landing real-world impact in research evaluation’, provides lots of interesting insights. In this blog we share our reflections and consider where Overton fits in.

For the report they surveyed 400 researchers, academic leaders, and heads of funding bodies to understand more about research impact evaluation systems. 

They set out to understand whether the current systems are fit for purpose, as well as how researchers and other stakeholders feel about them in order to explore whether there were ways that impact assessment could be improved.

The sector is currently going through a period of self-reflection when it comes to what is and isn’t working in research assessment, which is a good thing, with great work coming out of INORMS for example.

The opinions of citation providers may (rightly) be viewed with a sceptical eye here, but Elsevier’s reflections on the importance of getting research assessment right chimed with us. With public trust in science decreasing and researchers increasingly expected to justify spending by showing the economic and societal value of their work it’s especially important now.

The report includes interesting - and some concerning - statistics. 58% say they are frustrated by a lack of ways to show research’s impact on the wider world. 63% of researchers in a 2022 study said public scrutiny of research in general had increased since the pandemic. It seems that many researchers feel in an impossible position - held under a microscope, but struggling to find ways of demonstrating the real world value of what they do. 

The report shows a clear desire for change in the systems currently used to assess broader impacts, but acknowledges that there isn’t yet a clear community consensus on what should be included.

Elsevier does offer some suggested solutions. The report acknowledges that there’s no absolute model or solution to measure societal impact, but that proxy measures do exist - “indicators that give you an idea or signal about a possible impact” including proxies that can make links between research papers and policy documents. Elsevier’s Chief Academic Officer Nick Fowler identifies the United Nations Sustainable Development Goals (SDGs) as the best broader framework currently available to evaluate the connection between research and real-world impacts.

We agree with that, though mostly because (to paraphrase) it’s the worst framework except for all the others that have been tried from time to time.

They also reflect on how new technology can help create more useful metrics, noting that “data and text-mining techniques can reanalyse existing data sets to offer a richer picture of where and when impact occurs and how this links to the historic record of published academic papers”. 

The difficulty of gauging real impact via proxy measures is definitely something we grapple with at Overton. We know that the dynamics of policy influence are hard to track, let alone measure. However, it’s encouraging to hear that there’s a real enthusiasm for new ways to conceive of and assess impact within the research community. 

We know that simple policy citations are by no means the whole story. That's why we've dedicated considerable effort to enabling full-text search, finding mentions of academics by name, and policy to policy pathways, hopefully giving a more complete picture of how ideas and influence spread across various policy actors and sources. We also have a shiny new SDG filter that links policy documents to any of the related goals.