Dr Steven Hill, Head of Research, Higher Education Funding Council for England

Understanding the impact of the Research Excellence Framework

Steven Hill

Head of Research Policy

Science Summit 2014

03 December 2014

• Introduction to the Research Excellence Framework (REF)

• Results publication and analysis

• Analysis of REF submissions

• Evaluation of the REF

• Future REF

Summary

• Introduction to the Research Excellence Framework (REF)

• Results publication and analysis

• Analysis of REF submissions

• Evaluation of the REF

• Future REF

Summary

Overall process

• Universities makes submissions

• Submissions in 36 discipline-based Units of

Assessment

• Universities select which eligible staff to include in

submissions

• Submissions are assessed by sub-panels, one for

each Unit of Assessment

• Sub-panels are grouped into 4 main panels,

responsible for consistency

• Results provided as quality profiles

Overview:

The assessment framework

Overall quality

Outputs

Maximum of 4 outputs per researcher

Impact

Impact template and case studies

Environment

Environment data and template

65% 20% 15%

Overview:

• Introduction to the Research Excellence Framework (REF)

• Results publication and analysis

• Analysis of REF submissions

• Evaluation of the REF

• Future REF

Summary

• 18 December 2014: Results day!

• Overall quality profiles for each HEI/UoA

• And sub-profiles for outputs, impact and environment

• Analysis of results and process

• [Eligible staff numbers from HESA]

• Early January 2015

• Publication of submissions

• February 2015

• Panel and sub-panel overview reports

• REF Manager’s report

The REF results and other outcomes

• Introduction to the Research Excellence Framework (REF)

• Results publication and analysis

• Analysis of REF submissions

• Evaluation of the REF

• Future REF

Summary

Analysis of REF Impact case studies

6,975

• Work commissioned from Digital Science and Kings College London

• Two objectives:

• Publish case studies as a searchable, text-minable database

• High level synthetic analysis of case studies – what do they tell us about the impact of research?

• Published at end of March 2015

• Crucial evidence for spending review

• Database will lead to further detailed analysis

Analysis of REF Impact case studies

• What are the features of Units of Assessment that have scored very highly in the REF?

• Funding? Talent management? Strategic focus? Multi- and inter-disciplinary research?

• Does this vary by HEI type? Discipline?

• Commissioning shortly

• Publication in mid 2015

Factors leading to high performance

• Introduction to the Research Excellence Framework (REF)

• Results publication and analysis

• Analysis of REF submissions

• Evaluation of the REF

• Future REF

Summary

• Work commissioned from RAND Europe

• Phase 1 – evaluation of preparation of submissions

• Challenges, benefits and costs to HEIs

• Sample of 21 HEIs (35% of case studies)

• Phase 2 – evaluation of assessment process

• Strengths and weaknesses of the impact assessment process

• Panel members: survey, focus groups, interviews

• Publication late March 2015

Evaluation of impact element

• Work commissioned from Technopolis

• Estimate of costs to sector of the REF

• Specific focus on ‘individual staff circumstances’

• Benefits of REF to institutions

• Comparison to equivalent study following RAE2008

• [Impact element covered in impact evaluation]

• Publication late March 2015

Review of costs and benefits

• Written feedback sought in early 2014

• Responses from more than 100 HEIs

• Headline messages:

• Benchmarking, driver of quality

• Some aspects burdensome for institutions

• Positive effects of new staff circumstances measures

• Many positive views about impact (but issues with the detail)

• Publication of full analysis late March 2015

Feedback from participating HEIs

• Conferences and focus groups November 2014/January 2015

• User panel members

• Academic panel members (x2)

• Covering all aspects of REF (including contribution to impact evaluation and metrics review)

• Publication late March 2015

Feedback from panel members

• Introduction to the Research Excellence Framework (REF)

• Results publication and analysis

• Analysis of REF submissions

• Evaluation of the REF

• Future REF

Summary

• Independent panel chaired by Professor James Wilsdon

• Call for evidence – summary published

• 57% ‘sceptical’ about use of metrics

• Literature review

• Feedback from REF panel members

• Workshops

• ‘In metrics we trust?’

• ‘Metrics for all?’

• Metrics in the Arts and Humanities

• Emerging findings March 2015, publication Summer 2015

Review of metrics and research assessment

• Request from minister

• Informal consultation with

• Overseas stakeholders

• UK stakeholders (roundtable event)

• Open survey

• Advice to minister in next few weeks

International REF

• Most evaluation evidence complete March 2015

• Policy development/informal consultations Summer 2015

• Proposals and formal consultation Autumn 2015

Future REF – tentative timetable

• Introduction to the Research Excellence Framework (REF)

• Results publication and analysis

• Analysis of REF submissions

• Evaluation of the REF

• Future REF

Summary

Thank you for listening

s.hill@hefce.ac.uk

@stevenhill