Top Banner
Scoring Goals or Changing Scoring Goals or Changing the Game: the Game: What Impacts Should We What Impacts Should We Measure? Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium: ‘New Approaches to Assessing the Non- Academic Impact of Social Science’ London, May 12-13, 2005
23

Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Mar 28, 2015

Download

Documents

Jonathan Weeks
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Scoring Goals or Changing Scoring Goals or Changing the Game:the Game:

What Impacts Should We What Impacts Should We Measure?Measure?

Jonathan LomasCanadian Health Services Research

Foundation Presentation to ESRC Symposium: ‘New

Approaches to Assessing the Non-Academic Impact of Social Science’London, May 12-13, 2005

Page 2: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

My question is, are we having an impact?

Page 3: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Is Research Ready for Action?Is Research Ready for Action?Medline search (1966-02] to identify articles stating:

a. “need more research” or “need less research” Need more 161/162

Need less 1/162

b. “more questions than answers” or “more answers than questions”

More questions 163/166

More answers 3/166David, AS. BMJ 2002; 323:1462-3

Page 4: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Assessing the Impact of What?Assessing the Impact of What?The Research Produced?

A single research study published in a journalA summary of some research studies written in plain language and posted on a web siteA systematic review with targeted dissemination of key messages to potential usersA body of research knowledge developed and discussed face-to-face with potential users

Page 5: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Assessing the Impact of What? Assessing the Impact of What? (cont)(cont)

The Research Production Process?All the activities of a research commissioning or granting agency (including training)All the activities of a research production facility (e.g. institute, department, university)The activities of a potential research user organization and its staffThe entire ‘research regime’ in a country

Page 6: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Where I WorkWhere I Work

CHSRF’s mission:

“To support evidence-based decision-making in the healthcare system”

Our ultimate desired impact is cultural change in the research and healthcare systems

Changing the game, not just scoring goals

Page 7: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

The Discipline of Objectives & Logic The Discipline of Objectives & Logic ModelsModels

Yogi Berra on objectives:“If you don’t know where you’re going, you might not get there”

Lewis Carroll on logic models:“If you don’t know where you’re going, any road will take you there”

Page 8: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Decision MakersPolicy Makers

Organized Interests e.g. drug companies,

professional associations

Service Professionals

Managers

‘Client’ & Public

Decision Maker Diversity

Page 9: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Why Social Science isn’t IBM or General Electric

ResearchersUniversity-

based

Stakeholder-based

System-based

Management Consultants

Policy Makers

Organized Interests e.g. drug companies,

professional associations

Service Professionals

Managers

Patient & Public

Decision Makers

PR

OB

LEM

S

SO

LUTIO

NS

Page 10: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Decision Makers

Researchers

Evidence-Based Decision-Making

Research Funders

Knowledge Purveyors

Receptor Capacity

Critical Evaluation P

RO

BLE

MS

SO

LUTIO

NS

Funding and Training Vehicles

Priority-setting structures

ISSUES & PRIORITIES

PRIORITY

TOPICS

IDEAS

RESEARCH

EVIDENCE

Other InfluencesPersonal Experience,

Anecdote, Wants, Interests, Myths, Assumptions, etc.

Synthesis and Influence

Linkage and

Exchange

Page 11: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

CHSRF’s ObjectivesCHSRF’s Objectives

1. To increase health system decision-makers’ appreciation of the value of research

2. To increase the production of research relevant to the needs of health system decision- makers

3. To increase the availability and acquisition of needed research by health system decision-makers

4. To increase the appraisal and application of needed research by health system decision makers

Page 12: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Increased Appreciation of Value of Increased Appreciation of Value of ResearchResearch

Programs and activities:overall ‘linkage & exchange’ approachconsultations with users for prioritiescase study presentations of value

Measurable outcomes:# of decision-makers participating in researchexpenditures on research commissioning by system% of employees in system with research training

Page 13: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Increased Production of Relevant Increased Production of Relevant ResearchResearch

Programs and activities:priority theme-based program-fundingcommissioned syntheses on current issuesapplied training programsencourage university incentives for applied research

Measurable outcomes:amount of research in priority theme areasself-reported awareness/use of research syntheses# of graduates with applied research skills

Page 14: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Increased Availability/Acquisition of Increased Availability/Acquisition of ResearchResearch

Programs and activities:plain-language research summaries (1:3:25, Mythbusters)face-to-face exchanges on timely topicscreation/support of knowledge networkscreation/support for knowledge brokering

Measurable outcomes:self-reported awareness of disseminated researchself-reported follow-up contact with researchersself-reported use of web-based and other resources for research evidence acquisition (audit computer bookmarks)

Page 15: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Increased Appraisal/Application of Increased Appraisal/Application of ResearchResearch

Programs and activities:training users in research appraisal and applicationorganisational best practices in research use

Measurable outcomes:self-reported application of researchchanges in organisational structures and processes to better accommodate researchIncreased sense of ‘decision-certainty’ where synthesised research is available

Page 16: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

The Attribution ChallengeThe Attribution Challenge

What, if any, is/was the role of research versus all the other influences on behaviour?

Page 17: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Adapted from Philip Davies, 2005

Program or Intervention Effectiveness

Types of Research Evidence

Implementation Evidence

Organizational Evidence

Economic/ Financial Evidence

Ethics Evidence

Forecast Evidence

Attitudinal Evidence

ExperimentalQuasi-Experimental

Counterfactual

SurveysAdmin DataComparativeQualitative

Cost-BenefitCost-Effectiveness

Cost-UtilityEconometrics

ExperimentalQuasi-Experimental

QualitativeTheories of Change

Public ConsultationDistributional Data

MultivariateRegression

SurveysQualitative

Page 18: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Adapted from Philip Davies, 2005

Types of Research Evidence

Program or Intervention Effectiveness

Implementation Evidence

Organizational Evidence

Economic/ Financial Evidence

Ethics Evidence

Forecast Evidence

Attitudinal Evidence

ExperimentalQuasi-Experimental

Counterfactual

SurveysAdmin DataComparativeQualitative

Cost-BenefitCost-Effectiveness

Cost-UtilityEconometrics

ExperimentalQuasi-Experimental

QualitativeTheories of Change

Public ConsultationDistributional Data

MultivariateRegression

SurveysQualitative

ResearchEvidence

Page 19: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Adapted from Philip Davies, 2005

Combining Research and Colloquial Evidence for Information

ResearchEvidence

Professional Experience &

Expertise

Political Judgement

Resources

ValuesHabits & Tradition

Lobbyists & Pressure Groups

Pragmatics & Contingencies

Page 20: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

A Final Zany IdeaA Final Zany Idea

The ‘impact file’ - a deductive approach to assessing impact

Add photo of impact file

Page 21: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

What’s In the Impact File?What’s In the Impact File?

T F H M & Y D G W - testimonialsAltered career trajectory (researcher or decision-maker)Changes in similar organization’s programs & processes - ‘lateral impact’Better communication of research and its implications - disseminationAwareness of research by decision-makers - acquisitionChanged decisions based on research – applicationChanges in researchers’ or decision-makers’ structures and processes - ‘cultural change’

Page 22: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

Maybe Settling for Second Best Maybe Settling for Second Best is OK!is OK!Using an impact file is:

Motley, ad-hoc and not comprehensive or systematicBiased to individual not organizational responsesQualitative and potentially non-generalizable

But, as Churchill said of democracy:“Many forms of government have been tried … No one pretends that democracy

is perfect … Indeed, it has been said that democracy is the worst form of government, except all those other forms that have been tried from time to time.” House of Commons, 1947

Page 23: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:

THANK YOU!THANK YOU!

www.chsrf.caor

www.fcrss.ca