Cessda saw task4-6_haguefocusgrppresentation_0616

This project has received funding from the European Union’s Horizon 2020

research and innovation programme under grant agreement number 674939.

The Hague Workshop June 2016 - Neil Beagrie (Charles Beagrie Ltd)

Focus Group – The Value and Impact of (Social Science)

Research Data Infrastructure

Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark



Introduction (Neil Beagrie)

• 2008 -2011 Keeping Research Data Safe (KRDS 1/2/3) with UK

universities, data archives and OCLC Research as partners

Preservation Cost Model for UK Universities; Longitudinal cost data + Benefits

Framework; Benefits Analysis Toolkit + KRDS Factsheet & User Guide

• 2012 -2016 (Economic) Value & Impact Studies with John Houghton

Economic and Social Data Service; British Atmospheric Data Service; Archaeology

Data Service; Synthesis of 3 studies; European Bioinformatics Institute

• 2015-2017 CESSDA-Saw

Horizon 2020 European Project. The value and economic impact of social science data

services: funding advocacy toolkit.



Neil Beagrie (Charles Beagrie Ltd)

Task 4.6: Funding Advocacy Toolkit

Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark



CESSDA-SaW Task 4.6Capturing and communicating the value and economic impact of social science

data services. Develop a benefit/cost advocacy programme and supporting tools;

assembling an evidence base to support the negotiation with ministries and funding

organisations; support advocacy with other core stakeholders such as data creators and

data users.

Timetable

Milestone

number

Milestone name Due date

MS26 Survey to gather and

validate requirements

April 2016

MS27 Draft Toolset and testing Oct 2016

MS28 Focus groups and

case studies

Jan 2017

Early days…. First phase focus group today



Presentation Overview

• Measuring Value and Impact of Research Data

Infrastructure (briefly)

• ESDS Impact Study (briefly)

• User requirements survey (briefly)

• Toolkit ideas (briefly)

• Perception and discussion



Measuring Value and

Economic Impact of

Research Data

Infrastructure



European Research Data Open Access

fuller and wider access to scientific publications and data

can help to accelerate innovation (faster to market =

faster growth); foster collaboration and avoid duplication

of effort (greater efficiency); build on previous research

results (improved quality of results); involve citizens and

society (improved transparency of the scientific

process). What is at stake is the speed of scientific

progress and the return on R&D investment

(FAQ open access to publications & data in Horizon 2020)



European Research Data Open Access

Project results which are related to privacy, trade

secrets, national security, legitimate commercial

interests and to intellectual property rights shall not be

requested in open access mode. Additionally, any data,

know-how and/or information whatever their form or

nature which are held by private parties in a joint

public/private partnership prior to the research action

and have been identified as such shall also not fall under

such an open access obligation.

(FAQ open access to publications & data in Horizon 2020)



Buildings / Equipment

Grid / ICT Networks

Staff

Data

Intellectual Capital

Training & Skills

Human Capital

Technical / Organisational

Environment

Organisational Capital

Professional

Networks

Relationship Capital

Research Infrastructures and Impact: tangible and

intangible capital



Previous Work

Big Science and Innovation Study for BIS July 2013

• Desk review of c. 100 studies internationally;

• 3 studies highlighted to BIS as being particularly good

examples of ‘good practice’ in the measurement of

economic impacts:

• Berkeley Lab 2010

• Human Genome Project 2011

• Economic and Social Data Service 2012 (authors)



Best Practice from ESDS study• Applies range of

methods;

• Includes counter-

factual;

• Data collection

tailored to different

stakeholders:

depositors, users,

research, teaching;

• Data weighting -

survey value

responses weighted

to reflect the overall

pattern of use from

weblogs;

• Case studies/ KRDS

benefits illustrate

benefits and impact

pathways.



• Investment value: annual operational funding plus the costs that depositors face in preparing data for deposit and in making those deposits

• Use value: weighted average user access costs multiplied by the number of accesses

• Contingent value: the amount users are "willing to pay“ for or “willing to accept” in return for giving up access

• Efficiency gain: user estimates of time saved by using the Data Service resources

• Return on investment in the data: the estimated increase in return on investment in the data and services due to the additional use facilitated by the Data Service (counter-factual/”cost of inaction”)

Economic measures of value



• UK Data Archive one of the largest European social

science data archives: perhaps the largest

• UK Data Archive established for 40 years: has built up

collections and users over time

• Includes Government datasets: but not all in scope of the

ESDS study

• ESDS study: excluded undergraduates

• Only economic impact study for any social science

data archive to date

ESDS Study: Context



ESDS Study: Return on Investment (ROI)

Benefit/cost ratio of

net economic value to

ESDS operational costs

5.4 to 1

Increase in returns

on investment in data and

related infrastructure arising from

additional use facilitated by ESDS

(counter-factual)

up to 10 to 1



8%

13%

19%

20%

22%

17%

0%

5%

10%

15%

20%

25%

No change 1-10% 10-25% 26-49% 50-75% >75%

ESDS Study: Researcher Efficiency Gains



ESDS study: User Time/Money Savings

1%

10%

22%

22%

30%

32%

32%

39%

40%

58%

66%

81%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Other (Please Specify)

ESDS provided training

Time/resources in teaching/training

Data licences and charges from other

providers

User support materials and help desks

Re-using data avoiding additional

surveying and field testing

Single licence point for multiple data

types

Long-term preservation of data

Hosting and provision of data

Data beyond scope to collect myself

Data quality:preparation, validation and

documentation of data

Ability to find data from single point of

access



KRDS Benefits Framework• Framework arranged on 3 dimensions with two sub-divisions each; Pick list

of common generic benefits

• Individual benefits identified and assigned within this

Internal External

WHO BENEFITS?

Benefitfrom

Curation ofResearch Data



A4

Summary

–

Benefits

Summary

for ESDS



ESDS Study - conclusions

• Economic benefits exceed the operational costs

• A mix of qualitative and quantitative methods is important

• But having economic numbers is seen as critical

• Results active in later funding bids for c.10 years

• How to extend work for other social science archives?



CESSDA SaW Toolkit

User Requirements Survey



Survey Q1

IN WHICH COUNTRY IS THE DATA SERVICE BASED?



Survey Q2THE DATA SERVICE'S CURRENT STAFFING IS

APPROXIMATELY:



Survey Q3THE DEVELOPMENT OF THE DATA SERVICE IS:



Survey Q6CAN YOU GIVE EXAMPLES OF THE ARGUMENTS OR FACTS YOU USE IN

ADVOCACY TO MAKE YOUR CASE FOR FUNDING THE DATA SERVICE?

1

1

1

2

2

3

3

3

4

4

4

5

7

7

9

10

11

Big data

Comparison with other countries

Declarations of support from the community

Project funding criteria/obligations

case studies for use/impact

European funding opportunities

ESFRI/ National roadmap

Quality of resources for teaching

Quality of research data

Membership/legal obligations

Uniqueness/impact of no service

Standards, trust, reputation

Data sharing and collaboration

Wider Benefits e.g. evidence-based public policy

Benefits to Research community: Discovery and Access, support services

Open data/access/science

Data re-use is cost-effective

Arguments used in advocacy (themes)



Survey Q10HOW USEFUL WOULD YOU FIND THE FOLLOWING TOOLS IN BUILDING A CASE TO

DEMONSTRATE THE ACTUAL OR POTENTIAL IMPACT OF THE DATA SERVICE TO

POLICY MAKERS AND FUNDING DEPARTMENTS?



Toolkit Ideas



Toolkit Ideas

• Factsheets (costs, benefits, ROI)

• User Guide

• Surveys

• Case studies

• Extras (tbc)



Tool Effort Grading Levels

hours days months



Effort and Use Knowledge Pyramid:

Costs Example

Cost Modelse.g. KRDS, 4C, DANS

Cost Datae.g. KRDS costs survey, 4C Costs

of Curation Calculator

“Rules

of Thumb”(‘KRDS’ Law,

Kryder’s Law, etc)



Costs Rules of Thumb (1)

• “Kryder’s Law” – disk

storage roughly halving in

cost every year

(comparable to Moores

Law for processing

power)

• Rules for a prolonged but not eternal period of

time (“Laws”)

Kryder slowdown.

David Rosenthal.

Chart by Preeti Gupta at UCSC



Costs Rules of Thumb (2)

Ingest costs about half, preservation

about a sixth, access about a third, of

the data archive life-cycle costs

“KRDS Law”



Effort and Use Knowledge Pyramid:

ROI Example

Impact

Studies

Benefit-Cost Ratio

“Cost of Inaction”

(counter-factual)



Cost of Inaction (1)Academic Sociology research and teaching user (ESDS Report interviewee)

Thinking about the last time you accessed ESDS data how long do you estimate it took

you to find and access it?

Only one or two minutes...

If there was no ESDS and the government did not deposit datasets, his work could not be

done at all. The cost for each ‘one-off access would be substantial – several thousand

pounds... Calculating any access costs and the time taken to negotiate licences etc. he

estimated ‘maybe’ £5,000 - £10,000 (assuming access could only be given by the data

providers themselves)

…What would you do and how long would it take if ESDS did not exist? He calculated

three months – if you knew there was a dataset and you were trying to get access.



Cost of Inaction (2)

Wicherts and colleagues emailed requests for data from the corresponding authors of all 141 empirical articles that had been published in the most recent two issues of four American Psychological Association (APA) journals. Their aim was to re-analyze the data and assess the robustness of the research findings to outliers.

"6 months later, after writing more than 400 e-mails–and sending some corresponding authors detailed descriptions of our study aims, approvals of our ethical committee, signed assurances not to share data with others, and even our full resumes-we ended up with a meager 38 positive reactions and the actual data sets from 64 studies (25.7% of the total number of 249 data sets). This means that 73% of the authors did not share their data."

Wicherts et al.(2006) The poor availability of psychological research data for reanalysis. American Psychologist 61, 726–728.



John Houghton (Victoria University) + Neil Beagrie (Charles

Beagrie Ltd) 4 joint studies to date.

Methods applied to:

Economic & Social Data Service (ESDS)

Archaeology Data Service

British Atmospheric Data Centre

European Bioinformatics Institute

Value + Impact Analysis of Data Services



Perceptions and Discussion

Relevance? (Nationally)

Relevance? (CESSDA/Europe)

—thanks for your attention!

website: www.cessda.net / twitter: @CESSDA_Data