What have we learnt from REF2021? Five challenges for the next REF round

Policy impact is one of the most ‘efficient’ routes to REF stars - a fact which has not gone unnoticed by the Higher Education sector in the UK. In this blog, Overton reflects on the previous REF exercises, the challenges that persist and what the future might hold for REF2028 and beyond.

About the REF

For those unfamiliar, REF is the peer-review assessment of research in higher education institutes (HEIs) across the UK, run every four or five years. 

It provides a broad evaluation framework with three main elements: the quality of research outputs (accounting for 60% of the final evaluation), their impact (accounting for 25%) and the environment that supports it (accounting for 15%).

In the REF, impact is defined as an ‘effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’. The weighting of impact in this (2021) round of REF had increased by 5 percentage points compared to the 2014 round. 

As a result there has been an increased focus on the ‘impact agenda’ by funding bodies, reflected in grant applications and in project progress reporting requirements for awardees. This is a move which has been broadly seen as positive by academics, according to a 2021 UKRI review of perceptions of the REF amongst researchers. Overall, there has (as expected) been an increased uptake of the ‘impact agenda’, with an enhanced overall performance in the 2021 REF results. 

Challenge 1: Understanding what works

The race to find ‘what works’ in academic-policy engagement is now well underway, with academic-policy relations and consequently, impact, set to increase. This presents a new challenge - policy engagement has historically been a little like panning for gold. Research staff might participate in roundtables, phone calls, meetings and so on in the hope of achieving an output which is demonstrably ‘impactful’ or ‘REFable’, but this is not always the case. And, at the end of the day, whilst using research-based evidence seems like a no-brainer, one has to factor in the extraneous influence of other stakeholders and events on the MP making and implementing the policy. 

Challenge 2: Bridging the gap between the academic and policy worlds

Another challenge is the mis-match between the lead time facing policy professionals looking to develop and implement a policy, and that of securing research-based evidence which answers the question in its entirety. If the research exists, great, but how much time do civil servants actually have for going through numerous research articles? And if it doesn’t, they’re unlikely to have the time to wait for the policy objectives to be researched. One potential solution to this is for researchers and research development staff to factor policy impact in from the start. Citation and mention data can help to identify evidence gaps, inform strategies and provide evidence with which to approach policy professionals from the start of projects for a co-designed approach. This approach has the potential to ensure buy-in, and ‘as you go policy impact’ from the start. Another advantage of this approach is that it delivers insights which are actionable and therefore more likely to have a wider impact - though obviously, the time and money to deliver the research need to be considered also. One potential way around this is the ‘pump-priming’ of shorter term collaborative research projects or rapid evidence reviews - but these mechanisms are still perhaps in the ‘pilot’ testing phase.

Challenge 3: Varying interpretations of best practice

What is interesting is that there is no ‘mandate’ for how HEIs should approach policy impact in general. Sure, there’s emerging learning around ‘best practices’ when it comes to achieving policy impact, but not when it comes to identifying opportunities and demonstrating the success of these approaches in the sense of the REF.  Teams involved in delivering the REF within each university can have different expertise, sizes and approaches. This may mean that the ratio of REF staff to research staff can vary quite a bit, which may either be a barrier or an opportunity depending on context! 

Challenge 4: Capturing the breadth of policy impact

There is also quite a broad definition of ‘policy impact’ - sure, you can get a citation, but what about all the other forms that policy engagement, influence and impact can take? What about participation in a round table, as a member of an expert advisory group or media features? At the moment, these can often be submitted as evidence in research assessments, as well as testimonial letters from various organisations, but collating these links, screenshots, downloads and so on can be resource intensive - both in terms of time and money.

Getting to the crux of ‘demonstrating’ these kinds of impacts and enhancing understanding of what a fair approach to assessments might look like will be a key challenge to those in the Higher Education sector wishing to engage often time-poor colleagues in policy influencing activities who may need this ‘hard’ evidence to progress in their institutions and fields. 

Challenge 5: Collecting the right data and limiting the scope for bias

With such a nuance in understanding of both policy impact and what is ‘REFable’, there may also be concerns around biases - both in the type of engagement and in how it’s demonstrated. Whilst there is an increasing body of work around the use of technology and data to support and streamline these activities, there has yet to be an established or widespread shared understanding of what kinds of data can be consistently collected and used for this purpose - nor does it address the current challenge of particular sources of evidence outside of a citation or mention!

There may be some biases in how the information is collected, for example - Overton don’t currently cover all sources (though we can add specific sources if requested), nor do we cover all possible types of impact beyond citations and mentions, although this is something we would be really keen to work with stakeholders to explore, as we have on other aspects of the database. 

We wonder if data collation is the key - not necessarily just our database, but in the broader sense of reducing the administrative burden of impact evidence collation. By lightening the load and using data-driven insights to inform research development strategy, would we see an increase in the capacity and quality of policy engagement in REF2028 across the Higher Education sector? Would the ‘league table’ positions shift? 

Ideally this approach would go hand in hand with encouraging policymakers to play their part by developing widespread, publicly available means of acknowledging researchers in their work to feed into data collation.

Final thoughts

Whilst we’ve explored some potential approaches in this blog, it’s important to reflect on our definition of policy impact - not all academic-policy engagement delivers impact in the ‘traditional’ sense of a citation, but potentially an acknowledgement or mention of a particular individual or institution and some not even that. 

With this in mind, we think there is lots of scope for exciting developments in policy impact reporting, both in the UK and elsewhere! We hope Overton’s policy citation and mentions data can hopefully present some answers to these enduring challenges, but there is lots more to be done!