Program Evaluation Ralph Gonzales, MD, MSPH Professor of Medicine; Epidemiology & Biostatistics 20 May 2008.

Program Evaluation

Ralph Gonzales, MD, MSPHProfessor of Medicine; Epidemiology & Biostatistics

20 May 2008

Program Evaluation

Outcome Evaluation

Process Evaluation

Resource Evaluation

Relevant Perspectives

• Payor

• Organization

• Public Health/Gov’t

Program Evaluation Can Help To…

• Measure intervention’s effectiveness on targeted process or outcome measures.

• Determine most efficient and effective strategy for implementation of intervention

• Verify the mechanisms through which you believe your intervention is working

• Guide/support replication in other settings

• Align program goals with delivery system or stakeholder goals.

• Determine cost-effectiveness & priority

Pediatrics 120;2007:481-88… A “typical” Outcome Evaluation

Intervention Design: “Quality improvement collaborative”– Audit and feedback– Guideline dissemination/implementation– Buy-in/agreement to participate from CEO/CMO– Multidisciplinary team formation

• 4 state-wide “learning meetings”• Self-measurement activities• Monthly progress reports• One-on-one coaching calls

Outcomes– 13 hospital-based newborn preventive health care services

based on random sample of 30 medical records pre- and 30 medical records post-intervention

Vermont QIC Results

“Process Evaluation”: all Vermont hospitals participated; all formed teams; all attended each learning session; all participated in self-measurement, submitted monthly progress reports and participated in coaching calls.

Vermont QIC Conclusions• “QIC led to significant improvements in assessments for

breastfeeding adequacy, risk of hyperbilirubinemia, infant sleep position, and car safety seat fit”.

• “Costs estimated at $29,000 per hospital (not including hospital staff time)… cost per infant = $41

• “More research is needed to understand the predictors of a hospital’s success in improvement work”

What else would you like to know before trying this program in your state or hospital?

• What hospital infrastructure is needed in order to successfully participate?

• Will this program work outside Vermont?– Are there unique features of Vermont…?

• Who should pay for the program?

• Which program elements are essential?– Not enough variation to figure this out…

• What did each site actually DO?

Outcome Evaluation-Picking the outcomes

• Start with health/clinical outcome… work backwards to what processes or indicators are feasible, believable and compelling.

• Prioritize surrogate outcome measures– For surrogate measures, consider sensitivity

and specificity (or even validation)

• Specify a benchmark (outcome level) that would define a successful program

Handbook of Practical Program Evaluation JS Wholey, HP Hatry, KE Newcomer, eds. 1994

TABLE OF CONTENTS

Approaches to Evaluation Design1. Assessing feasibility and likely

usefulness of evaluation2. Designing and using process evaluation3. Using qualitative approaches4. Outcome monitoring5. Constructing natural experiments6. Convincing quasi-experiments: the

interrupted time series and regression-discontinuity designs

7. Ethical and practical randomized field experiments

8. Synthesizing evaluation findings

Practical Data Collection Procedures9. Use of ratings by trained observers10.Designing and conducting surveys11.The systematic use of expert judgment12.Acting for the sake of research: the use

of role-playing in evaluation13.How to use focus groups14.Managing field data collection from

start to finish15.Collecting data from agency records

Practical Data Analysis16.Using statistics appropriately17.Using regression models to estimate

program effects18.Benefit-cost analysis in program

evaluation

Program EvaluationData Collection Tree

Outcome Process Resource

Quantitative•Direct observation•Surveys•Chart review•Lab; Pharmacy•Payor admin. data

Qualitative•Direct observation•Focus groups•Personal interviews•Role play

Patients Staff Providers Managers Directors …

Chart Review Methods

Gilbert et al. Ann Emerg Med 1996;27:305-08.

Training -train with set of practice charts; describe in methods

Case selection -use explicit inclusion and exclusion criteria

Definition of variables -define key variables precisely

Abstraction forms -use standardized abstraction forms

Describe and ensure uniform handling of data that is conflicting, ambiguous, missing, or unknown. Use double-data entry

Meetings periodically review disputes and coding rules

Monitoring monitor performance (eg, occasional repeat abstractions)

Blinding blind abstractor to hypothesis, or at least group assignment

Interrater agreement test with 2nd rater… report kappa or ICC

Chart Review Methods

Gilbert et al. Ann Emerg Med 1996;27:305-08. %

Training -train with set of practice charts; describe in methods 18

Case selection -use explicit inclusion and exclusion criteria 98

Definition of variables -define key variables precisely 73

Abstraction forms -use standardized abstraction forms 11

Describe and ensure uniform handling of data that is conflicting, ambiguous, missing, or unknown. Use double-data entry

Meetings periodically review disputes and coding rules

Monitoring monitor performance (eg, occasional repeat abstractions) 4

Blinding blind abstractor to hypothesis, or at least group assignment 3

Interrater agreement test with 2nd rater… report kappa or ICC 5

Errors that can occur when transforming chart data to database

Gilbert et al. Ann Emerg Med 1996;27:305-08.

• Identification of clinical event• Select patients to be studied• Assemble charts• Locate desired information• Read note• Code information• Transfer data to computer database

Another Quantitative Program Evaluation…

Minimizing Antibiotic Resistance in Colorado (MARC)

Project Goals• Decrease community levels of antibiotic

resistance in Colorado• Decrease antibiotic use for viral respiratory

illnesses– Increase clinician knowledge of appropriate treatment

strategies for respiratory illnesses• Clinical practice guidelines & office based materials

– Decrease patient demand for antibiotics• Increase public and patient awareness about antibiotic

resistance and appropriate antibiotic use• Improve public self-care and when-to-seek-care strategies for

respiratory illnesses

A Social Ecological Model for Changing Patients' Expectations for and Use of

Antibiotics

Mediating FactorsIntrapersonal Level-awareness/knowledge-motivation (and past experiences)-skills and self-efficacyInterpersonal Level-changing social norms-new information in medical settingsCommunity Level-new information in mass media-new clinical practice guidelines

Level of Exposure to Intervention

noneSSCELSCEboth

CLINICAL ENCOUNTER

Patient Outcomes

antibiotic use

self care

no satisfaction

no illness duration

Patient Modifiers

demographysocioeconomicsinsurance typehealth statusmedical and treatment historyillness characteristics

7/01 11 7/02 11 7/03 11 7/04 7/05

SSCE develop implement reinforce

LSCE develop implement

MD-Ed reinforce reinforce

DIS

AbRx

Pub-S

Pat-S

MD-S

PRSP

Economic Analysis

Year 1 Year 2 Year 3 Year 4

THE MARC PROJECT TIMELINE

Interven

tion

Figure LegendSSCE: small-scale community education: household and office-based educational materialsLSCE: large-scale community education: television, radio, newspaper, and web site; development to include observation of

best practices and community focus groupsMD-Ed: clinician educational intervention: reinforcement via feedback of antibiotic prescription ratesDIS: dissemination activitiesAbRx: antibiotic prescription rates for pharyngitis, bronchitis derived from CMS Data Project MCOs and MedicaidPub-S: public and parent survey from intervention and control communities (Data Project and Medicaid households)Pat-S: pharyngitis and bronchitis patient outcomes survey from intervention and control community office practicesMD-S: clinician vignette study from intervention and control community office practicesPRSP: active invasive PRSP surveillance by Emerging Infections Program at Colorado State Health DepartmentEconomic Analysis: conducted in Year 4 with data from AbRx and Pat-S results from Years 1-3.

Evaluation

MEDIA IMPRESSIONSMEDIA IMPRESSIONS

PAID MEDIA billboards 2,319,800 bus shelters 64,715,400 bus tails 27,000,000 bus interiors 8,035,800Total OOH 102,071,000 Radio NPR 1,449,558 TV paid+bonus 491,382Total Broadcast 1,940,940Aggregate Total PAID 104,011,940

EARNED MEDIA TV 829,500 Radio 94,400Total Broadcast 923,900Total Print 3,505,400Aggregate Total EARNED 4,429,300

Figure 1A. In the past 3 months, have you seen or heard ads or

news telling you about antibiotic resistance?

0

10

20

30

40

50

60

Pre-BaselineWinter

Baseline Winter Mass MediaWinter

Perc

ent Res

pond

ing 'Yes

'

Mass Media

Comparison

P=0.04

Figure 1B. In the past 3 months, have you seen materials in a medical

provider’s office about problems with overusing antibiotics?

0

10

20

30

40

50

60

Pre-BaselineWinter

BaselineWinter

Mass MediaWinter

Perc

ent Resp

ondin

g 'Yes'

Mass Media

Comparison

P=0.03

-30

-25

-20

-15

-10

-5

0

5

10

15

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10

Net

Ant

ibio

tic P

resc

riptio

ns p

er 1

000

pers

ons

per

mon

th

General Population

campaign

2002 2003

Figure 2A. Antibiotic Prescriptions Filled by General Population in Retail Pharmacies

Mass Media Community Receives Fewer Antibiotics than Control Community

Mass Media Community Receives More Antibiotics than Control Community

P=0.30

-30

-25

-20

-15

-10

-5

0

5

10

15

11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10

Net

Ant

ibio

tic P

resc

riptio

ns p

er 1

000

pers

ons

per

mon

th

Total MCO Members

campaign

2002 2003

Figure 2B: Total Antibiotic Prescriptions Filled by Managed Care Organization Members

Mass Media Community Receives Fewer Antibiotics than Control Community

Mass Media Community Receives More Antibiotics than Control Community

P=0.02

-30

-25

-20

-15

-10

-5

0

5

10

15

11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10

Net

Ant

ibio

tic P

resc

riptio

ns p

er 1

000

pers

ons

per

mon

th

Pediatric MCO Members

campaign

2002 2003

Figure 2C: Antibiotic Prescriptions Filled by Pediatric Managed Care Organization Members

Mass Media Community Receives Fewer Antibiotics than Control Community

Mass Media Community Receives More Antibiotics than Control Community

P=0.01

-30

-25

-20

-15

-10

-5

0

5

10

15

11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10

Net

Ant

ibio

tic P

resc

riptio

ns p

er 1

000

pers

ons

per

mon

th

Adult MCO Members

campaign

2002 2003

Figure 2D: Antibiotic Prescriptions Filled by Adult Managed Care Organization Members

Mass Media Community Receives Fewer Antibiotics than Control Community

Mass Media Community Receives More Antibiotics than Control Community

P=0.09

-30

-25

-20

-15

-10

-5

0

5

10

15

20

25

11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10

Net

Offi

ce V

isits

per

100

0 pe

rson

s pe

r m

onth

Pediatric MCO Members

campaign

2002 2003

Figure 3: Office Visits by Pediatric Managed Care Organization Members

Mass Media Community has Fewer Office Visits than Control Community

Mass Media Community has More Office Visits than Control Community

P=0.01

Net savings in pediatric and adult prescription and visit costs attributable to the mass media intervention

Control Community Mass Media Community

Baseline year costs per

1000 members

Intervention year costs per 1000 members

Percent Change

Baseline year costs per 1000

members

Predicted* costs per

1000 members

Actual costs per

1000 members

Savings per 1000 members

Pediatric prescriptions

$37, 644 $43,239 +14.9% $34,257 $39,348 $33,810 $5,538

Adultprescriptions

$23,404 $23,948 +2.3% $25,527 $26,121 $24,180 $1,941

Pediatric visits

$4710 $5212 +10.6% $5,408 $5,983 $5,016 $968

Adult visits

$2168 $2075 -4.3% $2,133 $2,042 $2,116 $(73)

*Predicted costs based on percent change observed in the control community.

“Darn It! If only we had done a qualitative process evaluation…”

Questions we could have addressed• Confirm change in visit rates due to mass

media campaign… and patient driven (vs. physician driven).

• Parent experience with and reactions to mass media campaign

• Physician experience patients, and whether they perceived any changes in patient/parent expectations as a result of the mass media campaign.

Process Evaluation-describes how a program works

Process Evaluation-Goals

• Understand your results– Help explain heterogeneity in effects

• Allow replication

• Refine and improve (CQI)

Process Evaluation-Objectives

• Describe the intervention that got delivered

• Assess level of exposure to the intervention

• Assess the experience of those exposed to the intervention– Was the target audience engaged?– Did processes occur en route to the proposed behavior change?

• i.e., did knowledge, attitudes, proximal behaviors improve in a manner that led to an improved main outcome behavior?

• Are there system/organizational factors that modify the effect of the effectiveness of the intervention?

1. Measure Intervention “Delivery”

• Extent of intervention implementation– Interviews, questionnaires and surveys of

implementers and participants

– Stealth observers

2. Assessing Exposure to the Intervention

• Measure exposure periodically, continuously or retrospectively

• Use observation, self-report and/or existing data sources

3. Describing Experience of Those Exposed to Intervention

• Use self-report through interviews, focus groups or surveys

• Assess their experience with intervention, perceived barriers/facilitators that are most closely associated with failure/success

• Link factors to conceptual framework

Process Evaluation-Focus Group Guide

• What do patients, staff and clinicians consider to be strengths of the program?

• What do patients, staff and clinicians dislike about the program?

• What do patients, staff and clinicians recommend for improving the target behavior?

• Do personnel have adequate resources (money, equipment, facilities, training, time) to achieve program goals?

RE-AIM

Glasgow R et al. Am J Prev Med 2006;30:67-73.

• Reach (participation rate + representativeness)• Effectiveness (effect size… standardized effect

size if comparing interventions)• Adoption/Acceptability (participation rate and

representativeness of practices (or providers)• Implementation (consistency of intervention

delivery across practices/providers• Maintenance (sustainability of effect size).

Performance of DP and DHC on Individual RE-AIM Dimensions

Diabetes Priority• Computer-assisted health

behavior change program• T2DM action plan• Pts completed in office

prior to diabetes visit• Takes about 20-30 min

• Implemented by clinic staff at 30 Colorado mixed-payer PC practices (52 MDs, 886 pts)

• MDs review & endorse pt’s action plan

• Clinic staff discuss plan with pt and schedule f/u visits

• $222 per patient

Diabetes Health Connection• Computer-assisted health behavior

change program• T2DM action plan• Focused soley on diet/exercise• Separate visit with Health Coach• Takes 30-45 min

• Implemented by Intervention Team• 42 MDs, 335 pts @ mixed payer & HMO• $547 per patient

Devising a Process Evaluation for the IMPAACT Trial

Break-out Groups-15 min group meeting

-10 min group presentations-10 minutes wrap-up

Fill-In the BlanksProcess Evaluation

Quantitative Approach- Target group(s)- Data collection methods- Key questions to be addressed

Qualitative Approach- Target group(s)- Data collection methods- Key questions to be addressed

ED/Hospital

IMPAACT Trial

Clinical Practice Guidelines (national)

+

Performance Feedback (group)

+

Patient Education

IMPAACT Multi-Dimensional Intervention Strategy

• Each ED randomized to intervention received the following:1. Provider education (practice guidelines)

delivered by local opinion leaders– Opinion leaders attended “train-the-trainer”

session

2. Group audit and feedback

3. Patient education

• Sites provided individualized adaptation of components

Patient Education

• Waiting Room Patient Education– Pamphlets/Cards– Informational Kiosk

• Examination Room Materials– Bronchitis Posters

ABx Treatment of URIs/Bronchitis Decreased at Intervention Sites

Metlay et al, Ann Emerg Med, 2007.

Conceptual Framework

Things We DidPhysician Education• Seminars (b)• Guidelines (b)• Audit/Feedback (b)• Opinion Leader (b)Patient Education• Waiting Room Brochures (a)• Kiosk (a)• Exam Room Posters (a; b)

Patient FactorsSociodemographicsKnowledgeCase MixExpectations & Demands

Physician FactorsKnowledgeAttitudesStaff vs. MoonlighterType (attending; housestaff;

midlevel)Diagnostic Uncertainty

System FactorsVA vs. EMNetWait TimesFast Track/Urgent Care ZonesAccess to Primary Care; F/UCompeting InterventionsSecular Trends/ActivitiesQI Culture

Abx Rx Decision

Other Outcomes• CXR ordering• Return Visits• Patient Satisfaction

a

b

c

Site

Responder vs. Non-

responder

ARI Antibiotic

Prescription Rates

Year 1 (%)

ARI Antibiotic

Prescription Rates

Year 2 (%)

ARI Antibiotic

Prescription Rates

Year 3 (%)

Percent Change in Antibiotic Rx Rates

VANM

Non-responder

45 55 66+21

EMNM

Non-Responder (near goal) 39 33 33 -6

EMNY

Non-Responder

77 62 59

-18

VAILResponder

75 61 51-24

EMGAResponder

18 29 6-12

EMILResponder

51 29 22-29

VANYResponder

77 59 39-38

Percent change calculated as Year 1-Year 3; Responder is defined as a site that either met goal targets OR had a greater than 20% reduction in antibiotic prescription rates.

Local Project Leader Report Stealth Observer Report

OVERALL Rating

SiteKioskRating

PosterRating

OverallSelf-Report Rating

KioskRating

PosterRating

OverallStealth ObserverRating

*combines local opinion leader, stealth observer, andfocus-group/ interview data

VANM 1 1 0 2 2 2Fair

EMNM 1 2 1 0 0 0Weak

EMNY 1 1 1 0 0 0[RG1] Weak

VAIL 1 1 1 2 1.5 2Fair

EMGA 2 2 2 2 2 2Excellent

EMIL 1 2 2 2 2 2Excellent

VANY 1 1 1 2 2 2Fair

0=Poor1= Fair2= Excellent

Site

Responder vs. Non-

responder

ARI Antibiotic

Prescription Rates

Year 1 (%)

ARI Antibiotic

Prescription Rates

Year 2 (%)

ARI Antibiotic

Prescription Rates

Year 3 (%)

Percent Change in Antibiotic Rx Rates

OverallImplementation

Rating

VANM

Non-responder

45 55 66+21

Fair

EMNM

Non-Responder (near goal) 39 33 33 -6 Weak

EMNY

Non-Responder

77 62 59

-18 Weak

VAILResponder

75 61 51-24 Fair

EMGAResponder

18 29 6-12 Excellent

EMILResponder

51 29 22-29 Excellent

VANYResponder

77 59 39-38 Fair

Percent change calculated as Year 1-Year 3; Responder is defined as a site that either met goal targets OR had a greater than 20% reduction in antibiotic prescription rates.

TRIANGULATION

Site Visits-Qualitative Process Evaluation

• One-on-One Interviews– Site PI– Chief Quality Officer– Nurse Manager

• Focus Groups– ED nursing staff– Noon conference presentation, Q&A

Analyzing TransciptsLikert Scale Ratings

Physician Champion: 1= weak, unknown to participants or not seen as a leader 5= clear strong advocate; leader among peers; well respected by all

Patient satisfaction: 1=patient satisfaction does not appear to be valued and was not mentioned as a barrier to improving antibiotic use5= patient satisfaction is measured and reported at the provider level and appears to greatly influence decisions

QI Culture: 1= no experience with QI efforts or negative experience 5= significant positive experience with QI; involves all members of the healthcare team in a collaborative model

Sample QuotesPhysician Champion

PI :“They’ve been calling me the antibiotic [tyrant] around here for a while”

Nurse Manager:“ He is passionate on this” and “He talks in the ED and the community”

PI: “Because I became so heavily associated with antibiotic use and the appropriate use of antibiotics it comes up a lot of times just when we interact. There have been a number of times when physicians have come up to me and said, ‘I thought about you the other day when I had a patient with a cold and I didn’t give them antibiotics.’”

Site

Responder vs. Non-

responder

ARI Antibiotic

Prescription Rates

Year 1 (%)

ARI Antibiotic

Prescription Rates

Year 2 (%)

ARI Antibiotic

Prescription Rates

Year 3 (%)

Percent Change in Antibiotic Rx Rates

Overall QI

Rating

Overall Influence of

Patient Satisfaction

Rating

Overall Physician Champion

Rating

VANM

Non-responder

45 55 66+21

4.5 2 2.5

EMNM

Non-Responder (near goal) 39 33 33 -6 3.5 4 3

EMNY

Non-Responder

77 62 59

-18

2 4 2

VAILResponder

75 61 51-24

4 4.5 4.5

EMGAResponder

18 29 6-12

5 5 5

EMILResponder

51 29 22-29

3.5 4 5

VANYResponder

77 59 39-38

2 2 4

Percent change calculated as Year 1-Year 3; Responder is defined as a site that either met goal targets OR had a greater than 20% reduction in antibiotic prescription rates.

TRIANGULATION

Main Outcome Surrogate Outcome

Barton

Belkora

Bryant

Davis

Eaton

Flaherman

Guy

Jose

Kim

Kaimal

Nguyen

Shin

Tsui

Velayos