Viney2011

Capturing and analysing the impact of MRC funded research

Ian Viney

Head of Evaluation, MRC Strategy Group

March 2011

MRC Evaluation ProgrammeProgress so far

• Capture and analyse the results of MRC funding

– Develop approaches to assess progress, productivity and quality of MRC research output at an aggregated (portfolio) level

• Provide information to strengthen the case for medical research

– Develop quantitative and qualitative information about the impact of MRC research

MRC Evaluation ProgrammePriorities for 2011/12

• Strengthen strategy development and decision making

– Support discussions at Strategy Board, measure progress with the strategic plan

• Improve understanding of economic/academic/societal impact and what might lead to it

– Identify improved indicators, encourage research on the “science of science”

• Support partnership working

– Collaborate with other funding agencies internationally

OUTPUTS/OUTCOMES IMPACTS

• Generation of new knowledge

• Improvements to health (living longer and with better quality of life)

• Development of collaborative networks

• Improving the performance of existing businesses

• Leveraging funding • Attracting R&D investment from global business

• Dissemination of research • Improving public policy and public services

• Research materials • Creating new businesses

• Intellectual property/licensing

• Delivering highly skilled people to the labour market

• New products/interventions

• Academic impact (effects on further research including other disciplines)

• Changes to policy

MRC Evaluation Framework

INPUT

Fu

nd

ing

for

Researc

h a

nd

Tra

inin

g

MRC e-ValCapturing output/outcome/impact

• MRC e-Val is an online database used to gather information about outputs/outcomes/impacts arising from MRC research

• MRC e-Val is building a robust dataset of evidenced output which allows progress/productivity and quality of MRC research to be analysed, and this information to be utilised in decision making

• MRC e-Val is used each year to collect updated, structured feedback from researchers throughout the lifetime of their MRC funding, and beyond

• Take a holistic approach, include papers and patents, but aim for a much wider range of output types

• E-Val data can “set case studies in context” of the productivity and quality of all MRC output. Comprehensive data obtained across the whole MRC portfolio

• A baseline dataset against which progress and trends can be prospectively tracked over time

MRC e-Val 2010

• Feedback from all researchers that have held MRC funding since 2006 (3714 MRC awards to over 2800 scientists)

• Includes fellowships, grants and Unit/Institute programmes

• Significant new system developments between 2009 and 2010 have made data entry and administration of the system easier, speeding up data cleaning, analysis and publication of results

• Submitted responses for 91.4% of funding agreements (3339/3655) obtained in e-Val

• These responses correspond to 92.3% of MRC spend between 2006/07 and 2009/10 (£1.88/£2.0 billion in RDW)

• STFC and Chief Scientist Office (CSO) Scotland have implemented e-Val

• 37,500 publications reported (30,000 papers between 2006-2010)

• Details of 5,000 active collaborations involving 6,000 partners

• Over 200 published patents (roughly 30% are licensed)

• Over 50 spin out companies with an evidenced link to MRC research since 2006

• Over 130 citations in policy documents, including 30 citations in NICE guidelines since 2006

• 360 new products and interventions in development, around 40 launched onto the market since 2006, including around 10 new drugs

• £340m of inward investment to UK research and development from overseas and private sector research funding between 2006 and 2010

MRC e-Val 2010 dataset Example Outputs/Outcomes/Impacts

Impact Profile of MRC Publications 2006-2008(Rebased Impact to end 2009 Data & analysis: Evidence, Thomson Reuters)

Network analysis of collaborations within MRC Regenerative Medicine portfolio

MRC Research has contributed to the creation and growth of new companies

Examples of changes in policy/practice (2009)

• Martin Dennis, Edinburgh – findings of the CLOTS trial; stockings not effective at preventing venous thromboembolism in acute stroke patients - impact on NICE and international clinical guidelines and treatment affecting approximately 80,000 people a year in the UK

• Paresh Vyas, MRC Molecular Haematology Unit (British Journal of Haematology paper) recommended newborns with Down’s Syndrome should have a full blood count to screen for a preleukaemic condition – now most do (approximately 750 each year in the UK).

• Simon Thompson, MRC Biostatistics Unit, helped to develop the basis for the UK national screening programme for abdominal aortic aneurysms (AAA). AAAs affect 4% of men aged 65-74 (approximately 80,000).

• Ray Waters, University of Cardiff, is deputy chair of the Committee on Medical Aspects of Radiation in the Environment (COMARE) - which has advised on the risks of sunbeds and made recommendations regarding legislation for the use of sun parlours – the misuse of which is estimated to lead to 100 additional deaths a year from skin cancer in the UK.

Influences on policy/practicePathways to Impact

Research

paper

Citation in policy

document

Informs changes to

Clinical practice/

economy etc.

• ROD

• PubMed

• Over 200 guideline issuing organisations

• NLM/UKPMC digitising NICE guidance

• E.g. ERNIE database

• Researchers participate in policy setting processes and/or

• Research is cited in policy documents

Potential pathways

Specific pathway

Approaches for analysis

Products/interventions (2010)

Results from MRC e-Val http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/MRCe-Val2009/index.htm

Published summaries of 6 sections from MRC e-Val 2009 on the MRC website

Reports e.g. “Impact of MRC Research” used as briefing for BIS/new ministers early in 2010

University/MRC Unit/Institute “dashboard” contrasting local MRC e-Val results with the whole MRC portfolio

In addition to spreadsheets with all raw data, arranged by section and principal investigator

Capturing output in 2011/12

• MRC will continue to use MRC e-Val to gather evidence of progress, productivity and quality of MRC output. Data gathering is planned again for October 2011

• CSO (Scotland) is piloting e-Val and STFC are currently gathering data using e-Val

• NIHR have just completed an output survey using a different approach

• RCUK is re-developing ESRC’s “Society Today” system to collect data for AHRC, EPSRC, ESRC and BBSRC

• MRC is in discussion with medical research charities to pilot the next phase of output gathering approaches which will encourage adoption of a “federated” approach between output datasets

•Consistent time series for medical research funding in CVD and MH from 1975-1992

•Clear conceptual framework relating to GDP gain from “spillovers”

•Estimation from literature of the magnitude of this GDP return

•Development and application to CVD and MH, of a ‘bottom-up’ approach to estimate health gain in terms of QALYs

•Analysis of UK guidelines (5 CVD and 12 MH) to provide indicators of lags and proportion of benefits attributable to UK

•Suggestions for developing research agenda

•Strong quantitative argument for investment in medical research

Medical Research: What’s it worth?

(2008)