Planning, Implementing, and Evaluation Using the RE-AIM Framework with Samantha Harden, PhD

Planning, Implementation, and Evaluation Using the RE-AIM Framework

HPLive.org Presentation: March 13, 2015

Contributors: Russ E. Glasgow, PhD and Paul A. Estabrooks PhD

Presenter: Samantha M. Harden, PhD, Human Nutrition, Foods and Exercise, Virginia Tech

RE-­‐AIM  Dimensions   Example  Studies  History  of    

RE-­‐AIM    

Outline

Outline

“If we want more evidence-based practice, we need more

practice-based evidence.” Green LW. Am J Pub Health 2006

•  Internal  validity  perspec=ve  §  The  magnitude  of  effect  as  the  key  indicator  of  readiness  for  transla=on  and  adheres  to  the  principles  of  evidence  ra1ng  for  determining  efficacy  

•  External  validity  perspec=ve    § ACen=on  to  interven=on  features  that  can  be  adopted  and  delivered  broadly,  have  the  ability  for  sustained  and  consistent  implementa1on  at  a  reasonable  cost,  reach  large  numbers  of  people,  especially  those  who  can  most  benefit,  and  produce  replicable  and  long-­‐las1ng  effects  

Brief History of RE-AIM

Glasgow RE, Vogt TM, Boles SM. Evaluating the Public Health Impact…Am J Public Health, 1999;89:1322-1327

Original RE-AIM

•  First  published  ar=cles  in  1999  

•  Originally  intended  to  increase  balance  between  internal  and  external  validity  

•  First  used  to  evaluate  preven=on  and  health  behavior  change  programs  

•  RE-­‐AIM  Trivia:  was  going  to  be  called  ARIEM  

Gaglio B & Glasgow RE. (2012). Evaluation approaches for dissemination and implementation research. In: R. Brownson, G. Colditz, & E. Proctor (Eds.), Dissemination and implementation research in health…1st Edition pp. 327-356). New York: Oxford University Press.

6

www.RE-­‐AIM.org  

A CHOICE?

–  Program A: 16 session physical activity program that produces a 150 minute per week change in moderate intensity physical activity for 8 out of every 10 participants

–  Program B: 16 session physical activity

program that produces a 150 minute per week change in moderate physical activity for 2 out of every 10 participants

Which program do you think is better?

Answer: It Depends! � Why does it depend?

–  Who delivers? ÷ Program A: Trained master’s level health educators ÷ Program B: Administrative assistants in community health

center –  How easy is it to implement?

÷ Program A: Moderately difficult ÷ Program B: Moderately easy

–  What resources? ÷ Program A: Group exercise area and counseling rooms ÷ Program B: Email access and participants can do activities at

home or in neighborhood.

Which is Better?

Answer: It Still Depends! –  How scalable is it?

÷ Program A: 20 people can participate per class session which includes 90 minute counseling session and 3 one hour classes each week.

÷ Program B: 100 people can participate per session which includes monitoring of physical activity and sending out weekly newsletters.

–  What does it cost? ÷ Program A: 33 hours/week for 6 months from health educator

for every 16 successes (20 people per group). ÷ Program B: 8 hours/week for 6 months from administrative

assistant for every 20 successes. –  How sustainable are the effects?

÷ Program A: 50% return to baseline activity after 6 months ÷ Program B: 50% return to baseline level after 6 months

Which is Better?

RE-AIM: Goals for translating useful interventions into regular practice

10

•  Move from a paradigm that emphasizes: •  The magnitude of effect as the key indicator of readiness

for translation and adheres to the principles of evidence rating for determining efficacy

•  Move to one that emphasizes: •  Attention to intervention features that can be adopted

and delivered broadly, have the ability for sustained and consistent implementation at a reasonable cost, reach large numbers of people, especially those who can most benefit, and produce replicable and long-lasting behavior changes

•  Reach •  Effectiveness •  Adoption •  Implementation •  Maintenance

The RE-AIM Framework:

Glasgow et al, AJPH, 1999

What is the RE-AIM Framework?

12

Definition: The number, percent of target audience, and representativeness of those who participate.

Data Needed:

Denominator—number of eligible contacted for potential participation

Numerator—number of eligible that participate

Comparative information on target population

E-AIM ELEMENTS:

13

Example: Move More.

Data:

Denominator—Inactive or insufficiently active adults going to the doctor for a physical (n=1518 total; 607 eligible; 218 referred)

Numerator—number of eligible that participate (n=115)

Participation Rate: 115/607=19%

Almeida et al. JSEP 2005

E-AIM ELEMENTS:

14 Almeida et al. JSEP 2005

Enrolled Participants Census Tract Total

Sample (n=115)

Stimulus Control (n=44)

Standard Care

(n=71) Female 60.9% 65.9% 57.7% 51.1%

48.8 (±11.9)

48.6 (±11.3)

48.9 (±12.3)

30.4

White 58.9% 56.8% 60.3% 55.3% Black 22.3% 27.3% 19.1% 19.8% Latino 12.5% 6.8% 16.2% 14.8% Asian 0% 0% 0% 5.6%

E-AIM ELEMENTS:

15

Definition: Change in outcomes and impact on quality of life and any adverse outcomes

Data needed:

Primary Outcome

Quality of life

Potential negative outcomes

R -AIM ELEMENTS:

16

Example: Family Connections Data:

Primary Outcome: Significant reductions in BMI z-score.

Quality of life: Improvement in quality of life with lower weight status

Potential negative outcomes: No evidence of heightened eating disordered symptoms

Estabrooks et al. AJPM 2009 Shoup et al. QLR 2008

R -AIM ELEMENTS:

Definition: Number, percent and representativeness of settings and educators who participate.

Data needed:

Denominator—number of eligible sites contacted for potential participation

Numerator—number of eligible sites that participate

Comparative information on target population of sites

RE- IM ELEMENTS:

Example:

Data:

Denominator—105 counties in Kansas eligible to participate

Numerator—48 agreed; 48/105=46%

Representativeness—Less active agent, less likely to deliver; Smaller population counties, more likely to deliver

Estabrooks, Bradshaw, Fox, et al. , AJHP, 2004 Estabrooks, Bradshaw, Dzewaltowski , & Smith-Ray, ABM, 2008

RE- IM ELEMENTS:

19

Definition: Extent to which a program or policy is delivered consistently, and the time and costs of the program.

Data needed:

Information on program components and essential elements

Information on resource use

RE-A M ELEMENTS:

Example:

Data:

Variability in delivery of program components based upon local tailoring of the program.

On average, 80% of program components were delivered as intended

2.5 hours of delivery agent time per participant compared to 36 hrs per participant in control

RE-A M ELEMENTS:

21

Definition:

Individual/member target: Long-term effects and attrition.

Setting/educator: Extent of discontinuation, modification, or sustainability of program.

Data needed:

Primary outcome assessment 12 months post intervention

Documented sustained delivery

RE-AI ELEMENTS:

22

Data:

Decreased BMI z-scores sustained 12 months after intervention complete

Data:

RE-AI ELEMENTS:

Individual level factors balancing internal and external validity:

� Shift from focus on the numerator to the denominator � Generalizability to target population � Avoid contributing to disparities � Common comparison for decision making including

unintended consequences � Robustness when combined with adoption: what

works best for whom, and under what conditions

Summary: Key RE-AIM Issues to Improve Translational Research

Setting level factors balancing internal and external validity:

� Will the intervention fit in a typical practice setting? � Generalizability to who will deliver the program � Initial start-up and ongoing costs � Understanding structure and who makes adoption

decisions (and how they are made) � Characteristics of the intervention, setting, culture,

and organization that help or hurt implementation

Summary: Key RE-AIM Issues to Improve Translational Research

RE-AIM: Goals for translating useful interventions into regular practice

Develop and translate research and practice-based interventions that can:

be adopted and delivered broadly,

have the ability for sustained and consistent implementation at a reasonable cost

reach large numbers of people especially those who can most benefit,

produce replicable and long-lasting behavior changes

One more example!

Integrated Research-Practice Developed versus Pipeline Model in Physical Activity Programming: A comparative analysis. Samantha M. Harden, Sallie Beth Johnson, Fabio A. Almeida, Paul A. Estabrooks In Preparation

Adoption-Based Randomized Control Trial

Integrated Research-Practice Model Efficacy to Effectiveness to Demonstration to

Dissemination Model

Fit Extension Active Living Everyday

Untitled

Implementation

0

100

200

300

400

500

600

Fit Ex ALED

Cost assessed as implementation hours

•  Degree delivered as intended •  ALED ~90% •  Fit Ex ~ 80%

•  Adaptation •  ALED None reported •  Fit Ex Numerous small changes in feedback timing and

structure

Effectiveness

� Reach ¡  Fit Ex-75 participants per program

¡  ALED 15 participants per program

¡  Both underrepresented by men

� Maintenance- Delivered for 3 years post initial evaluation

Descriptive Information