Moving Beyond Simply Determining the Efficacy of …re-aim.org/wp-content/uploads/2016/06/Estabrooks_AHCQR...with your Target Audience and Stakeholders…from the outset to integrate

December 11, 2009

Moving Beyond Simply Determining the Efficacy of Behavioral

Interventions

Paul A. Estabrooks, PhDAssociate Professor

Human Nutrition, Foods, & Exercise

VT Center for Translational Obesity Research

The Community Guide• A method to move from research to

recommendations:

• Systematic Review of effectiveness, applicability, potential side effects, economic outcomes, and implementation issues.

• Takes steps to exclude studies that may limit the validity of the findings.

• Recommendations for Physical Activity, Nutrition, & Obesity

• “Beacon of Light to Community Practitioners”

2Briss et al. AJPM 2000; Kahn et al., AJPM 2002

Obesity Trends* Among U.S. AdultsBRFSS, 1985

(*BMI ≥30, or ~ 30 lbs. overweight for 5’ 4” person)

No Data <10% 10%–14%

Recommended Strategies by the Community Guide1980 to 1985:

Eight



No Data <10% 10%–14%


ElevenTotal to Date:

Nineteen



No Data <10% 10%–14% 15%–19%


Twenty-TwoTotal to Date:

Forty-one



No Data <10% 10%–14% 15%–19% ≥20%


Twenty-FiveTotal to Date:

Sixty-Six



No Data <10% 10%–14% 15%–19% 20%–24% ≥25%



No Data <10% 10%–14% 15%–19% 20%–24% ≥25%



No Data <10% 10%–14% 15%–19% 20%–24% ≥25%



No Data <10% 10%–14% 15%–19% 20%–24% ≥25%



≥30%No Data <10% 10%–14% 15%–19% 20%–24% ≥25%



≥30%No Data <10% 10%–14% 15%–19% 20%–24% ≥25%

What’s up with the evidence?

Lots of Information,Lost on Integration

13

• Community Guide PA Findings• Access & Outreach

• PA by 48%, Aerobic Capacity by 5%

• Social support interventions• PA by 44%, Aerobic Capacity by 5%

• Individually adapted interventions• PA by 35%, Energy expenditure by 64%

14

How about applicability?

Should be applicable, if adapted



Lack of Science and Operational Integration

If we build itthey will use

ScientistAs Expert

Efficacious Product

Intervention Development

Little Attention to Broader HealthPolicy and Cultural Context

Optimal Conditions

OrganizationClinic

Program Delivery

Staff

Delivery Site(s)

Driven by Broader HealthPolicy and Cultural Context

It doesn’t fit

Community Guide Evidence

• Physical Activity Systematic Review• 94 included out of 253 eligible

• Worksite Obesity Systematic Review• 20 included out of 31 eligible

• School Obesity Systematic Review• 10 included out of 31 eligible

• Primary reason for exclusion: Threats to internal validity

16

The Community Guide lets us know what is efficacious, but...

17

• we don’t know what works to help people be more active, eat more healthfully, and maintain a healthy body weight.

A couple of examples Efficacy Research may NOT Currently

Translate into Practice:

Application of evidence-based SAMSA “model” program based on efficacy RCT completely ineffective in applied context*

“efficacy trials conducted by developers provide insufficient evidence of effectiveness”

Replication of same smoking cessation program found efficacious in efficacy trial did not work—even in exact same setting with same patient group—in effectiveness study.**

* Hallfours et al, Am J Public Health, 2006;96:2254-2259

** Stevens, Glasgow, et al, Medical Care, 2000;39:451-459

• Which of the following is better?• Program A: 16 session weight loss

program that produces a 5% loss for 8 of every 10 participants (i.e., average weight loss 4%).

• Program B: 16 session weight loss program that produces a 5% for 2 of every 10 participants (i.e., average weight loss 1%).

Expanding Metrics

• How about these?• Program A: 33 hours of staff time

for each 5% loss.

• Program B: 6.5 hours of staff time for each 5% loss.

Expanding Metrics

• Or these?• Program A: Needs extensive

training to deliver.

• Program B: Can be delivered by anyone.

Expanding Metrics

• Or these?• Program A: 25% of those who lose

weight keep it off for a year.

• Program B: 50% of those who lose weight keep it off for a year.

Expanding Metrics

• Expanding metrics beyond efficacy and effectiveness

• Evidence that is contextual, practical, & robust

• Re-consider the appropriate philosophy of science• isolating, simplifying, and holding conditions • Studying programs in context and the

impact of different contextual factors

Doing things differently

Glasgow, Society of Behavioral Medicine Debate on Evidence, 2007

Combining metrics for interventions with the potential to influence the public health

24

• Reach large numbers of people

• Be widely adopted by different settings

• Be consistently implemented by staff members with moderate levels of training and expertise

• Produce replicable and long-lastingeffects (and minimal negative impacts) at reasonable cost

Glasgow, Society of Behavioral Medicine Debate on Evidence, 2007

25

12/11/2009 8:58:03 AM

Total population of students in the State

Total population of students in the schools that adopt the intervention

Total population of students in the schools with trained and certified physical education teachers

Total population of students who go to the after school programs

Of those who got the intervention the number it worked for

Of those who it worked for the number who sustained the change

ADOPTION

IMPLEMENTATION

REACH

EFFECTIVENESS

MAINTENANCE

Estabrooks, American College of Sports Medicine Resource Manual for Graded Exercise Testing and Prescription, in press

Program Application

Estabrooks et al., Annals of Behavioral Medicine, under review

27

12/11/2009 8:58:03 AM

Adoption

0

20

40

60

80

100

Number of Counties Participating

20022003200420052006


28

12/11/2009 8:58:03 AM

Reach

0

5000

10000

15000

20000

25000

2002 2003 2004 2005 2006

Year

Num

ber o

f Par

ticip

ants


Effectiveness & MaintenanceTime points & Comparisons

Baseline Activity Status

Active Insufficiently Active

Inactive Significance

BaselineModerateVigorous

277.8163.0

64.87.5

0 0

* p < 0.001; A > INS > IA * p < 0.001; A > INS > IA

8-WeeksModerateVigorous

282.7164.7

166.8 67.8

203.3 58.4

* p = 0.001; A > IA > INS* p < 0.001; A > INS > IA

6 Month follow-up

ModerateVigorous

225.4 110.9

171.8 51.2

134.1 39.2

p = 0.259* p = 0.002; A > INS > IA

Implementation• Determine Community Task Force Use

• Fidelity to Protocol for

• Subcommittees• Meeting Schedules• Strategy completion

• Compared with & without task force on specific strategies, reach, and effectiveness

0

10

20

30

40

50

60

70

80

Yes No

Task Force Use

0255075

100125150175200225250

Task No Task

Reach by Implementation


31

12/11/2009 8:58:03 AM

Maintenance-Organizational

0

5

10

15

20

25

30

35

40

45

0 1 2 3 4 5

Perc

ent o

f Cou

ntie

s

Number of Years Program Adopted

Why did it work?

32

Systems thinking allows researchers a view into critical delivery issues that allows the idea generation, planning,

and evaluation process to make evidence contextual, practical, and

robust for that system

Peer SharingGroup feedback

Sense of DistinctionGroup goal setting

Group Roles

Tested in

Team-building PA Interventions

Diverse Samples

Multiple Settings

Freq

uent

Con

tact

Research Staff D

elivery

Critical Elements

Scheduling & Cost of Delivery

OrganizationExtension

Office

Agents

Delivery Sites

Spac

e Li

mits

Limited Staff Tim

e

Office Staff Engagement

Cooperative Extension

Available Resources

Cur

rent

Hea

th P

rogr

ams

Demonstration Project

AppropriateFor Question

FitWalk Kansas

Re-invention of intervention retainingcritical elements but

reducing contact

Estabrooks et al., ABMEstabrooks & Glasgow, AJPM, 2006

Conclusions• Effectiveness is only part of the picture

• Focus on the Denominator (of settings, practitioners, participants)

• Plan for Generalization

• Partner with your Target Audience and Stakeholders…from the outset to integrate program, policies, & procedures in delivery contexts

• Everything is Contextual (customize and document it)

Klesges LM, et al. Ann Behav Med 2005;29:66S-75S; Estabrooks & Glasgow, AJPM: 2006; Green LW & Glasgow RE. Evaluating the Relevance, Generalization…Evaluation and The Health Professions 2006;29(1):126-153.

Real Science for Real People…

• The future is multiple (conditions, behaviors, interactive modalities)

• The future is complex (and we ignore complexity at our peril)*

• All models (and designs) are imperfect** – and greater tolerance, respect, and creativity is needed

• We need to UN-learn much of what we have been taught to answer the tough questions

* Glasgow RE, Emmons KM. Annual Review of Public Health Dec 6, 2006 epub** Sterman JD. Syst Dynam Rev 2002;18:501-531

Questions…

37

System Thinking Still needs to Fit in Broader Context

Walk Kansas

& Personality

BEHAVIOR

ValuesEvaluations

Bonding

Sense ofSelf

SocialSkills

SelfDetermin-

ation

SELFEFFICACY

SocialCompetence

Decisions/Intentions

ReligionCulture

PerceivedNorms

Motivationto Comply

Others'Beh&&Atts

SOCIALNORMATIVEBELIEFSContext

Social

DNA

ATTITUDES

InformationalEnvironment

KnowledgeExpectancies

EvaluationsValues

©2001 BrianFlay

Moving Beyond Simply Determining the Efficacy of …re-aim.org/wp-content/uploads/2016/06/Estabrooks_AHCQR...with your Target Audience and Stakeholders…from the outset to integrate

Documents