EVALUATION OF SOCIAL IMPACT

EVALUATION OF SOCIAL IMPACT

productive interactions in health (policy) research

HTAi conference Bilbao, 27-06-2012Jack Spaapen

2

Three questions (SIAMPI project)

Q1. What is social impact? - Conceptual and practical problems

Q2. What are ‘productive’ interactions? And what do they have to do with

social impact?

Q3. How to evaluate social impact: PI, intermediate impacts, social impact

2

3

Q1. WHAT IS SOCIAL IMPACT

- Conceptual problems- Linear models vs network approaches- Temporality- Attribution / contribution- Overlap between various impacts- Positive / negative impacts

- Practical problems- Reward systems- Indicators - Data collection

3

4

where is social impact?

4

5

research is contextualized

Topsector Health / Life sciences (Netherlands)

New Dutch Science Policy Top sectors: energy, water, food/agriculture,

health/life sciences, creative industry, …

Healthy aging (EU)Creative industry, new media, (serious) gamesNanotechnologyGreen meat production in 2020 in NLWater management (New Orleans)

Social impact is the outcome of a joint effort in a network of relations: research, industry, government, NGOs, consumer-organisations public private partnerships

Which data are relevant? Activities in the network coalitions, goals, interactions, intermediate results

5

6

New coalitions for Research, development and innovation

- Triple helix, golden triangle: research, industry, society (government, NGOs, general public), new collaborative arrangements: PPP (transdisciplinary research, transepistemic communities – Knorr-Cetina, science 3.0 - Miedema)

- Interdisciplinary input from research: natural science/technical fields, social sciences, humanities (technical knowledge, content, use/behavior)

- But also input from other expertise: politics, law, economists, ethics, consumer-interests, etc.

- Co-creation of ‘ new knowledge’ and practical solutions = innovation, research by design, iterative process (non-linear) -

- Consensus about long term goals (“healthy aging”, “clean energy”), but in the meantime shifting coalitions, different partners, different intermediate goals, different interests

- Problems arise from not being used to work together (academics – industry, not knowing what to formulate as research questions), institutional problems, political and cultural problems….

7

RESEARCH, DEVELOPMENT and INNOVATION (RDI) AS A LONG TERM NETWORK EFFORT

The network consists of a variety of

stakeholders working on a common

problem; but goals and people shift

Everybody produces knowledge,

everybody does research:

transdisciplinary collaboration

The result is to be socially robust knowledge : scientifically reliable, socially valuable

7

8

Q2. PRODUCTIVE INTERACTIONS

What are ‘productive’ interactions? - Relations in a network that produce something of value- Not only of economic value (knowledge-new technology-cash value)- Also socio-cultural, ethical, intellectual, technological, environmental

And what do they have to do with social impact?- They show what it takes to achieve impact- Guideline for indicators of intermediate impact

8

9

• Personal interactions : joint projects, advisory, consultancy, double functions, mobility

• Media interactions : • Texts : articles, books, catalogues, protocols, new diagnostics• Artifacts : instruments, exhibitions, models, designs

• Support: contracts, subsidies, patenting, licensing, sharing of people and facilities

• Instances of social impact

SIAMPI: productive interactions between science and societynetwork oriented

10

DATA AND INDICATORS: FOCUS ON INTERACTION AND INTERMEDIATE OUTCOME

representing productive interactions between a variety of stakeholders

= research, policy, industry, society

Data collection:(i) personal interactions(i) interaction thru media(iii) financial or material support(iv) intermediate output (v) indications of social impact (vi) new products, procedures, etc

10

11

Q3. HOW TO EVALUATE

CONCEPTUAL [self evaluation reports]• Mission orientation• Network analysis (thru texts, people, organisations)• Stakeholder approach (involvement from the start)

PRACTICAL• Network indicators - indications• Intermediate indicators – 3-5 years• Impact indicators – instances of impact

11

12

HEALTH CASE: NIVEL and LUMC

NIVEL public/private institute for health policy and primary health care

LUMC academic department (public health, gynaecology,

anatomy/embryology)

NIVEL: top down organisation of network activities, stakeholder contacts actively organised to safeguard financial support and to enhance chance of implementation of results, include all types of stakeholders

LUMC: bottom up, incentive structure based on WoS, but on the move from traditional academic department to contextual research (top sector policy for life sciences and health research)

EXAMPLE OF CASE STUDIES

Social impacts related to research at NIVEL or LUMC are often too intricate to be attributed directly to or identified with specific groups.

LUMC: Anatomy, stem cell projects. Results / impacts are only to be expected in the very long run (10-15 years)

In this case, we did see some impact in clinical practice of a neighbouring field, cardiology.

NIVEL: the result of one project in Public Health pointed out that currently popular health centers specifically for elderly only raised health care demand but did not improve the health situation of elderly.

Impacts (i) can be positive or negative; and (ii) take shape in various contexts: the academic group (also other groups!), the hospital, the policy domain (stem cells!)

13

14

Type of interaction

Example Effects (success)

Direct Consultation rounds with stakeholders Adaptation in research agenda

User groups, supervising boards (PPP consortia)

Adaptation of research projects

Presentations to health care professionals (conferences, meetings, post-academic teaching)

Sustaining relations with stakeholders, knowledge transfer

Collaboration in research Mutual adaptation in research projects

Indirect Annual plans Mutual agreements with funding agencies over future research agenda

Implementation plans Mutual agreements with funding agency over future implementation of project results

Reports & medical guidelines, scientific publications

Knowledge transfer, Response (uptake of knowledge) by a wider variety of stakeholders than those involved in direct interactions

Financial Contracts, Licences, Project grants Enabling completion of research projects

Lump sum grants Enabling independent research

15

SOCIAL IMPACT OF NIVEL: reception of reports per domain©AD PRINS

Google searches for reports / PDFs of 8 NIVEL domains

16

SOCIAL IMPACT OF LUMC: MEDICAL GUIDELINES FOR GPS© AD PRINS

Google searches for three guidelines of LUMC Dep. of Public Health and General Practice

Table 1: SIAMPI indicators for productive interactions

personal interactions between stakeholders

interaction between stakeholders through media

Financial / material interaction between stakeholders

•face-to-face meetings•double functions, other mobility arrangements•phone conferences•email•social media•videoconferencing•public debate•radio, tv, internet•etc.

•academic journals•professional journals•non academic journals•popular media•exhibitions•artefacts, models•films•master theses, graduate projects•standards, protocols•social media•etc.

•research contracts, public and private, and mixed, national, international•facility, instruments sharing• start ups•contribution “in kind” (people)•IPR arrangements, patents, licenses•Professional training•Other stakeholder interest•etc.

18

What is necessary to evaluate research in context

o A network perspective, mutual learning, extended review (more than peers), new reward systems, administrators that dare to care

o Development and use of new type of criteria and indicators that focus on process and intermediate results

o Continuous feed back between stakeholders, impact through learning!

o Use / acceptance of these methods at all levels of research (institute, national, EU)

19Research domain

nano ict Health care ssh

Country NL, France UK, NL, EU NL Esp, UK

Research type Frontier, basic, strategic

Basic, applied, TD strategic, applied, policy

Basic, strategic, applied

Research mode Academic, in collaboration with industry

Open to partnership of knowledge producers and users

Academic, open to collaboration with industry, govern-ment, patient groups, professio-nals

Academic, open to collaboration with policy, institutions, wider public, industry

Productive Interactions

Public understanding, ethical debates, policy making, products

Transport use, security, interaction between citizens and government

Consultation, colla-boration, regula-tions, proto-cols, commercial exchanges, PPPs, post academic training, patient organizations

Informal links and advice, formal research contract and collaborative projects, consultancy, cultural events

Social Impact Health, safety, public acceptance of nano tech

Transport use, security, interaction between citizens and government

Diagnostics, treatments, safety, general health, policy advice

Policy tools and techniques, management methods, cultural goods and services