Liam Cleere University College Dublin’s Senior Manager for Research Analytics and Reporting

Impact: how should we

capture it?Impact and the Humanities

June 2015

Liam Cleere, University College Dublin

2

Agenda

• Context

• Beyond Publications

• Measures of Impact for Arts & Humanities?

• Implementing Impact

3

About UCD

• Ireland’s largest university– over 1,300 faculty and 30,000 students

• €430m annual turnover

• National leader in research funding– €100m annual research income

• 139th in QS World University Ranking 2014/15

• Ireland’s Global University– home to over 6,000 international students from 124 countries and delivers degrees to over

5,000 students on overseas campuses

4

International research excellence

Top 50 in world - 2 Veterinary Science (40)Development Studies (45)

Top in Ireland - 12

Veterinary Science Development StudiesArchitecture/Built Environment Business & ManagementLaw Engineering -ChemicalEngineering – Civil & Structural Earth & Marine ScienceAgriculture & Forestry Materials ScienceStatistics & Operational Research Mathematics

Top 100 in world – 9English Language & Literature HistoryModern Languages Agriculture & ForestryVeterinary Science Geography & Area StudiesDevelopment Studies LawPolitics & International Studies

Source: QS Subject Rankings 2015

5

Irish Context

• 2006: Strategy for Science Technology and Innovation (SSTI)

– Double the number of PhDs

• 2009: Research impact through bibliometric analysis of research publications in peer reviewed journals

– Indicators of economic impact and societal contribution absent

• 2011: Research Prioritisation process: Increased focus on economic and societal impacts

– Government and funding agency led

– Focus on jobs, FDI and applied research

– SFI & Horizon 2020 require impact plans

• No National Research Excellence Framework

Beyond Publications Project

6

1. Current State Assessment 2. Future Options

3. Final Report Delivery &

Recommendations

2013 2014

Phase 1 Phase 2 Phase 3

Report:

• “Furthering the research impact of University

College Dublin”, May 2014

Steering Committee

Meeting

Steering Committee

Meeting

Steering Committee

Meeting

7

Why capture impact?

1. Higher Education Institutions overview – To enable research organisations to monitor and manage their performance and understand

the contribution that they are making to communities

2. Accountability– To demonstrate the value of research to government, stakeholders, and the wider public

3. Inform funding– To understand the socio-economic value of research and subsequently inform funding

decisions

4. Impact Journey– To understand the method and routes by which research leads to impacts, optimising the

potential of research findings and developing better ways of delivering impact

8

Point of view from ‘Furthering Impact’

• Broader role for university– Building a more just, inclusive and wiser society

• No standardised approach for addressing the

broader picture of research impact– provides an opportunity for UCD to clarify its position on research

impact, and

– implement an effective system for capturing research outputs and

communicating their value and relevance in social, cultural and

economic ways

9

CompositionDesign

PerformanceSoftware

ExhibitionArtefact

Digital or visual mediaDevice and product

*OtherScholarly edition

Patent/ published patent applicationConfidential report for external body

Website contentResearch datasets

BookWorking paper

Edited bookResearch report for external body

Book ChapterConference contribution

Journal article

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%1%

5%9%

11%11%11%12%

14%16%

18%21%

26%31%

42%46%

52%74%

77%96%

100%

Which of the following outputs result from your research activities?

Key outputs currently captured

CompositionDesign

PerformanceSoftware

ExhibitionArtefact

Digital or visual mediaDevice and product

*OtherScholarly edition

Patent/ published patent applicationConfidential report for external body

Website contentResearch datasets

BookWorking paper

Edited bookResearch report for external body

Book ChapterConference contribution

Journal article

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%1%

5%9%

11%11%11%12%

14%16%

18%21%

26%31%

42%46%

52%74%

77%96%

100%

10

Key outputs are being lost

Outputs beyond publications

CompositionDesign

PerformanceSoftware

ExhibitionArtefact

Digital or visual mediaDevice and product

*OtherScholarly edition

Patent/ published patent applicationConfidential report for external body

Website contentResearch datasets

BookWorking paper

Edited bookResearch report for external body

Book ChapterConference contribution

Journal article

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%1%

5%9%

11%11%11%12%

14%16%

18%21%

26%31%

42%46%

52%74%

77%96%

100%

11

Key outputs are being lost

Outputs beyond publications• Impact Case Studies• Research sample repositories• Written commentary and advice to agencies

and NGOs• Health education materials• Booklets, DVDs, Video• Public outreach (through seminars)• Public lectures / speech• Radio talk• Guidelines• Materials for museums• Tours• Advice• Field schools• Podcasts• Television• Popular magazines

12

Why UCD need to do something

• UCD bias towards publications as indicators of research excellence continues Bibliometric analysis remains the yardstick

• UCD Research Reputation suffers By not communicating research impact as a key building

block of reputation

• UCD cannot fully communicate the value & relevance of research to funders & other key stakeholders

• UCD falls further behind in telling its research impact story Compared to leading international institutions

13

What is impact?

• “the consequences of an action that affects people’s lives in areas that matter to them”1

1 ESF (2012) The Challenges of Impact Assessment (Pubd online) Available at: <http://www.dfg.de/download/pdf/dfg_im_profil/evaluation_statistik/programm_evaluation/impact_assessment_wg2.pdf> [Accessed 30 Mar 2014]

14

Distinction between Academic and Economic & Societal impacts

15

UCD’s Academic Impact

16%Above world average in 2010-2014?

Source: Elsevier SciVal

16

CollegeNo. Publication in

RMSNo. Publications

in Scopus

Approx. % Coverage in

ScopusScience 5,052 3,733 73.9%Health Sciences 3,714 3,257 87.7%Engineering & Architecture 3,068 1,951 63.6%Agriculture, Food Science & Veterinary Medicine 2,831 1,903 67.2%Human Sciences 2,430 883 36.3%Arts & Celtic Studies 1,367 199 14.6%Business & Law 1,309 564 43.1%Grand Total 19,771 12,490 63.2%

Coverage Issues

Source: Elsevier SciVal & UCD Research Management System 2010 to 2014

17

What gets measured gets done?‘The REF Effect’

Name Overall 2008 2009 2010 2011 2012 2013

% Increase 2008 to

2013University of Leicester 445 34 29 40 96 108 138 305.9%University of Liverpool 384 29 30 31 67 117 110 279.3%Queen's University Belfast 343 28 31 31 51 101 101 260.7%Trinity College Dublin 298 26 32 22 65 74 79 203.8%University College Dublin 348 36 35 50 60 83 84 133.3%

Increasing volume of journal articles in Arts and Humanities per year

Source: Elsevier SciVal

18

Difficulty of measuring book to book citation

• Prestige of the publisher of the book?

– e.g. Danish BFI list of publishers

• Paperback publication of what was originally a

hardback monograph?

• Coverage of books in Scopus continues to grow

—with a plan to reach 120,000 titles by the end

of the year

19

Altmetric for institutionsA measure of societal impact?

• UCD’s highest scoring publication is from the College of Arts & Celtic

CollegeNumber of articles mentioned

in social mediaNumber of articles

in Altmetric for Institutions

By total number of

authorsCollege of Science 1,110 4,619 415College of Health Sciences 1,006 3,901 345College of Agriculture, Food Science and Veterinary Medicine 365 1,761 162College of Engineering and Architecture 124 1,559 139College of Human Sciences 227 1,175 188College of Business & Law 52 586 88College of Arts & Celtic Studies 43 222 66

coverage in Altmetric for Institutions is problematic due to its bias toward counting references with Digitial Object Identifiers

20

What about Economic & Societal

impact?

21

Many ways to measure impact

• Input Measures

– Include research funding, human resources, existing knowledge, equipment and facilities.

• Output Measures

– refer to the measurement of the products of the research activity. The most obvious output measure is publications, but processes or tools used to

disseminate research can also be considered as a type of output indicator

• Expert reviews

– assesses research impact by obtaining information from groups of experts, for example the UK REF uses expert panels to rate the submissions from

universities

• Surveys

– structured surveys can be used as an impact assessment measure. An example of this approach is the Impact Finder Tool, created by the RAND

Corporation.

• Case Studies

– contain a narrative describing research impact with supporting metrics and references to detail the nature and scale of the impact.

• Hindsight studies

– attempt to retrospectively trace an observed research impact to the research inputs, activities, output and outcomes that led to the impact.

• Economic models

– are often used to assess value for money. Econometric analysis is used to assess research impact at a macro-level while cost-benefit analysis is often

used to determine impact from research projects or programmes.

22

Impacts linked to inputs, activities, outputs, outcomes and time

23

IMPLEMENTING IMPACT

24

Context for implementation

• UCD’s Strategy 2015-2020

– Increase the quality, quantity and impact of our research, scholarship and innovation

– Provide an educational experience that defines international best practice

– Consolidate and strengthen our disciplines

• Schedule• Resourcing plan• Governance structure

Draft Impact Implementation Roadmap

Optimise discoverability of publications

• Prepare for publication• Dissemination

approaches• Keeping Track of Your

Research

2. Enhance Academic Impact

3. Capture & Convey Socio-Economic Impact

2015 2016 2017 2018

Capture & convey impact & outputs beyond publications

• Implement Impact System (RMS II)

• Impact collection (using Seed funding round)

• Capture case studies• Classify• Assess• Reward

• Publish impacts

Phase 1 Phase 2

4. Evaluate impact

Creating impact from knowledge

• Academic impact assessment

• Economic Value added (EVA)

• Societal impact assessment

Phase 3

1. Excellence & Impact Framework

Definition & development of the impact & excellence framework

• Requirements• Key Performance Indicators

Planning

25

26

Research Quality Assessment Pilot

• Assessment of the quality and level of research activity in

the School of Archaeology benchmarked internationally

• Provide an overview of how research relates to T&L and

the student experience within the discipline

• Make recommendations to strengthen the discipline

• Inform strategic planning

27

The Process – Outline

• School Research Statement Report

• Individual Staff submissions (4 research outputs)

– Jan 2010-Dec 2014

• Impact Case Study/Studies

• Peer Panel Assessment – site visit 14-15 October

• Panel Report - December

28

Forming the overall quality profile

In forming the overall profile, the panel will assess three

sub-elements:

• Environment (15%)

• Outputs (65%)

• Impact (20%)

29

Research Statement Report

• Environment

– Staff FTE & Headcount

– Research Income €

– Contextual narrative on strategy, organisation, infrastructure, funding, collaboration

• Outputs

– 4 publications per staff

– Benchmarking outputs against peer institutions (incl. citation analysis)

– Research PhD numbers

• Impacts

– Narrative on context for impact

– Impact case studies

30

Collection of Outputs

• 4 per person (2010 -2014) using template

• Outputs can be:– Publications: [including books, book chapters, special issues; journals, monographs,

conference papers, policy evaluation reports, critical review articles etc.]

– Products: [this may include - physical artefacts, digital artefacts, research datasets, software

etc.]

– Patents

• Make outputs available online (where possible)– Institutional Repository / Research Information System

– Online Journal etc.

31

Output Collection Template

32

Capturing Impact

• Title of case study

• Images

• Summary

• Research description

• Details of the impact

• References

33

Timeline

- Collect 4 outputs per academic staff- Collect Impact Case studies

2. Collect Outputs

3. Finalise Documentation

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

4. Peer Review & Final Report1. Preparation

- Planning- Document process- Draft Research Statement Report- Explore indicators

- Finalise Research Statement Report- Finalise Outputs

- Review panel site visit - Final Report

1st April:Staff Briefing

30th June:Deadline for collection of outputs & case studies

1st Sept:Deadline for Finalised Research Statement Report

34

APPENDICESExamples of impact

35

History and Policy papers bring past issues to current

policy debatesHistory Hub – Connecting Past and Present

 History does not repeat, but it does rhyme – Mark Twain

What is the impact of pay-TV on sport? Why did the Irish health service develop in the way

that it did? How did small, local hospitals become so important? And was the policy focus misplaced on hospitals rather than on people’s health?

Would the 1950s style of bureaucratic diversity have prevented the ‘groupthink’ that became characteristic of the Central Bank and Department of Finance before the economic crash of 2008?

Did British policy responses to parades disputes during the 1990s peace process militate against a resolution of disputes which continue today?

Image: Postcard of St. Patrick’s Ward, St. Vincent’s

Hospital by paddykilty.wordpress.com

History Hub.ie is based at the School of History & Archives in UCD. It

hosts and podcasts, digitised archival documents and papers written by

historians on current affairs.

Access to the archival material gives people an understanding of the

process of history writing. HistoryHub.ie goes further, delivering open

access, peer reviewed research by leading historians in a format that is

robust, topical, and accessible, thus bringing history to the modern day.

Access to such a wide source of materials (archival documents, policy

papers, podcasts, original documentary videos) means history is no

longer a ‘closed book’.  The site offers a unique opportunity to analyse

the past and, most importantly, to learn from it. Active citizenship must

be informed citizenship. HistoryHub.ie remains central to discourse in

Ireland’s modern democracy.

36

Global crisis At the forefront of environmental humanities

What is the relationship between mud-slides, poverty, and economic policy in the Philippines?

What impact has the de-regularisation and

privatisation of ‘natural’ commodities such as water and oil had on our world and its environment?

In Ireland, how has the financial crisis and the

housing collapse changed our living spaces and surroundings?

Image: Pyrite Hell by Paul Reynolds   

UCD is at the forefront of interdisciplinary research examining the intersections between environmental and economic crisis, and in particular, capitalism’s transformations of cultural, environmental, and social landscapes.

Led by Dr Sharae Deckard from the School of English, Drama, and Film, UCD hosted a major academic symposium on ‘World-Ecology, World-Economy, World-Literature’ in 2013, which considered issues such as food and water security, energy regimes, disasters, and extreme weather.

In partnership with Dublin City Council, Dublin UNESCO City of Literature, and Cultúr Lab, the symposium was accompanied by a number of events open to the general public, including a photographic exhibition entitled ‘Landscapes of Crisis’, poetry readings, and discussion evenings.

37

War: The Facts Ireland’s first Centre for War Studies in UCD

According to the Peace Pledge Union, in recorded history since 3600 BC, over 14,500 major wars have killed close to four billion people – two-thirds of the current world population.

Since 1495, no 25-year period has been without war.

One out of every two casualties in war is a civilian caught in the crossfire.

When it comes to war studies, we learn about past atrocities to prevent history repeating itself. Founded in 2008 and led by Professor Robert Gerwarth, the Centre for War Studies in UCD promotes a wide range of international research activities focused on the origins, nature, and consequences of all war-related violence, from ancient times to the present day.

The Centre attracted the first ERC Grant ever in the Humanities in Ireland. It employs research staff from eight different countries and its output is translated into 25 languages.

38

Culture and Heritage at UCD

Source: The Economic, Cultural and Social Impact of UCD