Scoring Goals or Changing Scoring Goals or Changing the Game: the Game: What Impacts Should We What Impacts Should We Measure? Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium: ‘New Approaches to Assessing the Non- Academic Impact of Social Science’ London, May 12-13, 2005
23
Embed
Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Scoring Goals or Changing Scoring Goals or Changing the Game:the Game:
What Impacts Should We What Impacts Should We Measure?Measure?
Jonathan LomasCanadian Health Services Research
Foundation Presentation to ESRC Symposium: ‘New
Approaches to Assessing the Non-Academic Impact of Social Science’London, May 12-13, 2005
My question is, are we having an impact?
Is Research Ready for Action?Is Research Ready for Action?Medline search (1966-02] to identify articles stating:
a. “need more research” or “need less research” Need more 161/162
Need less 1/162
b. “more questions than answers” or “more answers than questions”
More questions 163/166
More answers 3/166David, AS. BMJ 2002; 323:1462-3
Assessing the Impact of What?Assessing the Impact of What?The Research Produced?
A single research study published in a journalA summary of some research studies written in plain language and posted on a web siteA systematic review with targeted dissemination of key messages to potential usersA body of research knowledge developed and discussed face-to-face with potential users
Assessing the Impact of What? Assessing the Impact of What? (cont)(cont)
The Research Production Process?All the activities of a research commissioning or granting agency (including training)All the activities of a research production facility (e.g. institute, department, university)The activities of a potential research user organization and its staffThe entire ‘research regime’ in a country
Where I WorkWhere I Work
CHSRF’s mission:
“To support evidence-based decision-making in the healthcare system”
Our ultimate desired impact is cultural change in the research and healthcare systems
Changing the game, not just scoring goals
The Discipline of Objectives & Logic The Discipline of Objectives & Logic ModelsModels
Yogi Berra on objectives:“If you don’t know where you’re going, you might not get there”
Lewis Carroll on logic models:“If you don’t know where you’re going, any road will take you there”
Decision MakersPolicy Makers
Organized Interests e.g. drug companies,
professional associations
Service Professionals
Managers
‘Client’ & Public
Decision Maker Diversity
Why Social Science isn’t IBM or General Electric
ResearchersUniversity-
based
Stakeholder-based
System-based
Management Consultants
Policy Makers
Organized Interests e.g. drug companies,
professional associations
Service Professionals
Managers
Patient & Public
Decision Makers
PR
OB
LEM
S
SO
LUTIO
NS
Decision Makers
Researchers
Evidence-Based Decision-Making
Research Funders
Knowledge Purveyors
Receptor Capacity
Critical Evaluation P
RO
BLE
MS
SO
LUTIO
NS
Funding and Training Vehicles
Priority-setting structures
ISSUES & PRIORITIES
PRIORITY
TOPICS
IDEAS
RESEARCH
EVIDENCE
Other InfluencesPersonal Experience,
Anecdote, Wants, Interests, Myths, Assumptions, etc.
Synthesis and Influence
Linkage and
Exchange
CHSRF’s ObjectivesCHSRF’s Objectives
1. To increase health system decision-makers’ appreciation of the value of research
2. To increase the production of research relevant to the needs of health system decision- makers
3. To increase the availability and acquisition of needed research by health system decision-makers
4. To increase the appraisal and application of needed research by health system decision makers
Increased Appreciation of Value of Increased Appreciation of Value of ResearchResearch
Programs and activities:overall ‘linkage & exchange’ approachconsultations with users for prioritiescase study presentations of value
Measurable outcomes:# of decision-makers participating in researchexpenditures on research commissioning by system% of employees in system with research training
Increased Production of Relevant Increased Production of Relevant ResearchResearch
Programs and activities:priority theme-based program-fundingcommissioned syntheses on current issuesapplied training programsencourage university incentives for applied research
Measurable outcomes:amount of research in priority theme areasself-reported awareness/use of research syntheses# of graduates with applied research skills
Increased Availability/Acquisition of Increased Availability/Acquisition of ResearchResearch
Programs and activities:plain-language research summaries (1:3:25, Mythbusters)face-to-face exchanges on timely topicscreation/support of knowledge networkscreation/support for knowledge brokering
Measurable outcomes:self-reported awareness of disseminated researchself-reported follow-up contact with researchersself-reported use of web-based and other resources for research evidence acquisition (audit computer bookmarks)
Increased Appraisal/Application of Increased Appraisal/Application of ResearchResearch
Programs and activities:training users in research appraisal and applicationorganisational best practices in research use
Measurable outcomes:self-reported application of researchchanges in organisational structures and processes to better accommodate researchIncreased sense of ‘decision-certainty’ where synthesised research is available
The Attribution ChallengeThe Attribution Challenge
What, if any, is/was the role of research versus all the other influences on behaviour?
Adapted from Philip Davies, 2005
Program or Intervention Effectiveness
Types of Research Evidence
Implementation Evidence
Organizational Evidence
Economic/ Financial Evidence
Ethics Evidence
Forecast Evidence
Attitudinal Evidence
ExperimentalQuasi-Experimental
Counterfactual
SurveysAdmin DataComparativeQualitative
Cost-BenefitCost-Effectiveness
Cost-UtilityEconometrics
ExperimentalQuasi-Experimental
QualitativeTheories of Change
Public ConsultationDistributional Data
MultivariateRegression
SurveysQualitative
Adapted from Philip Davies, 2005
Types of Research Evidence
Program or Intervention Effectiveness
Implementation Evidence
Organizational Evidence
Economic/ Financial Evidence
Ethics Evidence
Forecast Evidence
Attitudinal Evidence
ExperimentalQuasi-Experimental
Counterfactual
SurveysAdmin DataComparativeQualitative
Cost-BenefitCost-Effectiveness
Cost-UtilityEconometrics
ExperimentalQuasi-Experimental
QualitativeTheories of Change
Public ConsultationDistributional Data
MultivariateRegression
SurveysQualitative
ResearchEvidence
Adapted from Philip Davies, 2005
Combining Research and Colloquial Evidence for Information
ResearchEvidence
Professional Experience &
Expertise
Political Judgement
Resources
ValuesHabits & Tradition
Lobbyists & Pressure Groups
Pragmatics & Contingencies
A Final Zany IdeaA Final Zany Idea
The ‘impact file’ - a deductive approach to assessing impact
Add photo of impact file
What’s In the Impact File?What’s In the Impact File?
T F H M & Y D G W - testimonialsAltered career trajectory (researcher or decision-maker)Changes in similar organization’s programs & processes - ‘lateral impact’Better communication of research and its implications - disseminationAwareness of research by decision-makers - acquisitionChanged decisions based on research – applicationChanges in researchers’ or decision-makers’ structures and processes - ‘cultural change’
Maybe Settling for Second Best Maybe Settling for Second Best is OK!is OK!Using an impact file is:
Motley, ad-hoc and not comprehensive or systematicBiased to individual not organizational responsesQualitative and potentially non-generalizable
But, as Churchill said of democracy:“Many forms of government have been tried … No one pretends that democracy
is perfect … Indeed, it has been said that democracy is the worst form of government, except all those other forms that have been tried from time to time.” House of Commons, 1947