Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation

Designs and Types of Evidence for Dissemination and

Implementation Research C Hendricks Brown

University of Miami Miller School of Medicine

Director, Center for Prevention Implementation Methodology

Director, Prevention Science and Methodology Group Director, Social systems Informatics

chbrown@med miami edu

1

Acknowledgements

• Center for Prevention Implementation Methodology (Ce-PIM) for Drug Abuse and HIV Sexual Risk Behavior (NIDA) PI C Hendricks Brown

• Advanced Center to Improve Pediatric Mental Health Care (P30 MH074678) aka Implementation Methods Research Group (IMRG) PI John Landsverk

• Scaling up MTFC in California NIMH: R01MH076158, PI Patti Chamberlain

• Methods for MH/DA Prevention and Early Intervention, NIMH/NIDA R01-MH040859, PI C Hendricks Brown

• Collaborative Synthesis of Adolescent Depression Trials, NIMH R01-MH040859-

22 PI C Hendricks Brown • NIDA R01 DA019984 A Follow-up of Classroom Services to Prevent Drug Use,

PI Jeanne Poduska,

• NIDA R21DA024370, Scaling-up Prevention Services for Early Drug Abuse Risk in School Systems, PI Jeanne Poduska

2

Acknowledgements Co-Authors John Landsverk, Rady Children’s Hospital Patricia Chamberlain, Oregon Social Learning

Center Lawrence Palinkas, USC Wei Wang, University of South Florida Lisa Saldana, Center for Research to Practice Jeremy Goldhaber-Fiebert, Stanford Sarah Horwitz, Stanford

3 NIH Training Institute for D&I Research Ce-PIM

Outline I. Factors Affecting Design of the D & I Study II. Three Dimensions for Typing Study

Designs for Implementation Research Illustrations - Microsimulation

III. Increasing the “Evidence” in Implementation Research

Roll-Out Randomized Implementation Trials Example of the CAL-OH Randomized

Implementation Trial IV. Summary

4 NIH Training Institute for D&I Research Ce-PIM

Where We Are in the Science of Implementation

• Just beginning • All of this is in flux • Role of Methodology Similar to That for

Prevention Science 20 Years Ago IOM Preventing Mental, Emotional, & Behav Disorders 2009

5

I. Factors Affecting Design of the D & I Study

• Mutually Agreed Upon Research Question

6

System to Support Adoption

1. What designs are used in implementation studies?

Changing Research Questions Effectiveness vs Implementation

7

Intervention Intervention

System to Support Adoption

NIH Training Institute for D&I Research Ce-PIM

General Influences on the Design of a Study

NIH Training Institute for D&I Research Ce-PIM 8

Determine Research Question(s) Research Team

Institutional/Community other Stakeholders & Partners

Intervention Conditions / Implementation Strategies

Population

Measures and Follow-up/ Assessment Procedures

Sampling, Sample Size and Enrolling

Assignment to Intervention Condition / Implementation Strategies

Ethics/Human Subjects/Values Financial/Logistic

s

Policy Makers, Funders

Handy Terminology

• Implementation (or Dissemination) Strategy – system-level program:

Glisson’s ARC Hawkins Communities that Care Community Development Team (CiMH) • Intervention – evidence-based program

9

Illustrative Classes of Research Questions for D&I Research and Common Types of Designs

1. What System Level Factors Impede or Facilitate implementation?

Descriptive Epidemiologic - Cross-Sectional Do More Evidence-Based Organizations

Adopt? 2. What implementation strategies lead to faster uptake, successful adaptation and sustained effects on target population? Interventional (Testing) -- Experimental 3. How does a Strategy Affect Implementation? Etiologic (Mediational) -- Multilevel Longitudinal

10

II. Three Useful Dimensions for Typing of Studies for D & I Research


How “Real World”

How much control over Implementation

Assignment What Levels Are Measured?

II. A Three Useful Dimensions for Typing of Studies for D & I Research


How “Real World” In Vivo -- full implementation as it takes

place In Vitro – implementation facilitated by

researchers In Silico –studies involving simulation

Examples • In Vivo : CMHS/SAMHSA Implementation

of the Good Behavior Game in 24 School Districts

Very Modest Federal Funding Districts Responsible for TA • In Vitro : IES/NIDA Funded Professional

Development Trial in Houston – Poduska PI Training, Supervision Supported through Research

• In Silico: MicroSimulation of Expected Impact of Project KEEP for a State Child Welfare System

13

Microsimulation

• Start with completed efficacy or effectiveness trial data of an intervention

• Model how this intervention would behave in a larger system if implemented broadly

• Design – Needs: 1. Individual level data from trial 2. Longitudinal data from a large system 3. Programming to model and extrapolate

findings

14

KEEP Efficacy Trial - Chamberlain

• Study design: Randomized controlled trial • Population: Children in 700 foster homes in

San Diego county (ages 5-12) • Outcome: Efficacy of a foster parenting

intervention (KEEP) in reducing placement changes over 200 days

• Findings: – Positive exits (reunification with biological families,

adoption, etc) increased – Negative exits (other foster homes, group care, etc)

decreased 15

KEEP Applied to Statewide Data – Jeremy Goldhaber-Fiebert

16

-0.60

-0.50

-0.40

-0.30

-0.20

-0.10

0.00

0.10

0.20

0.30

Effe

ct o

f KEE

P on

cu

mul

ativ

e in

cide

nce

Positive exits Negative exits Lateral moves

II. B Three Useful Dimensions for Typing of Studies for D & I Research


How much control in implementation

strategy and/or intervention assignment Research Influence Observational - NONE Quasi-Experimental SOME QI-Designs Randomized - Randomizes

implementation strategy to systems (components)

Standard Names for Experimental and Quasi-Experimental Designs (Cochrane,

EPOC)

• Randomized Controlled Trial (RCT) – random allocation to implementation/intervention arms

• Controlled Clinical Trial (CCT) – assigned in some controlled way (even-odd)

• Controlled Before and After Study (CBA) – Pre-post data same time for different arms Comparable controls More than 1 site in each arm • Interrupted Time Series (ITS) – sites

change from non-intervention to intervention 18

Biglan et al., AJCP 2006


Other Communities


Important Research Questions:


Which communities/organizations get invited an

— What percent of those invited participate? — What are the characteristics of

participants? — What are barriers to participation in this context?

Useful Nonrandomized Designs for Dissemination RE-AIM components


1. Broadcast of an Intervention Standardize Invitation and See Who Comes Kellam SG, Branch J, Brown CH, Russell G. Why teenagers come for treatment: A ten-year prospective

study of Woodlawn. Journal of the American Academy of Child Psychiatry 20:477-95, 1981.

2. Narrowcast of an Intervention Social Marketing to a Targeted Audience

3. “Timecast” of an Intervention Multiple Baseline design = Interrupted Time Series o Repeated measures over time of a community outcome o Introduce the intervention to a community midway through o Check whether community outcome differs before and after introduction

Hawkins NG et al., AJPM 2007


II. C Three Useful Dimensions for Typing of Studies for D & I Research


What Levels Are Measured? System Leaders County MH Agency Leader Implementation Agents Agency Hiring Clinicians Intervention Agents Clinicians Delivering Services Intervention Target Families Receiving Services

Example

25

Four Levels of Change for Assessing

Performance Improvement

Assumptions about Change

Larger System / Environment

Organization

Group / Team

Individual

Reimbursement, legal, and

regulatory policies are key

Cooperation, coordination, &

shared knowledge are key

Structure and strategy are key

Knowledge, skill, and

expertise are key

From Shortell, 2004

Types of Evidence for Implementation


• Formal Causal Inference • Hierarchy of Evidence

Generative Modeling

III. Increasing the “Evidence” in Implementation Research

• Evidence is Elusive Concept 1. Formal use of the causality in

philosophy/statistics 2. Intuitive Ideas: Ordering -- Hierarchy of evidence 2 trials with replicated results

better than 1 trial

27

Hierarchies of Evidence


• Preventive Task Force Expert Opinion … Pre-Post Design … Single Randomized Trial Multiple Randomized Trial

Caution: Hierarchy Not Always True


• Date from “Observational Studies” May Be More Valuable than that From Randomized Trials

Example: Do antidepressants cause suicide in

youth? RCT’s – no suicides occurred, few

attempts

Intuitive Idea About Evidence • Study 1: Implementation Strategy A significantly better

than Strategy B Is A truly better than B? Other Explanations -- imbalanced assignment -- differential support staff… • If we can modify Study 1’s Design in such a way to

– reduce the risk of other explanations – Leave all other factors the same

• Then the new Study Design has Stronger Evidence

30

Example

• Study 1: Offer Implementation A to 40 Counties and Compare the 23 Who Select A against the 27 Who Do Not Select A (B).

• Study 1’ : Randomly assign 20 of the 40 Counties to Receive A and 20 to Receive B. And All 40 Counties Do Adhere to this Strategy.

31

Roll-Out Randomized Implementation Trials

• Straightforward way to improve the evidence over interrupted time series or multiple baseline designs

• Generally Acceptable by Communities • Beneficial properties in statistical power

and logistics Brown et al. Clinical Trials 2006 Brown et al. Ann Rev Pub Health 2008

32

Biglan et al., AJCP 2006


Inferential Challenge with this Design


• What if and Exogenous Factor Happens at one of these Times of Transition?

• What if you Select the Most Promising Communities to Work with First?

• What if there are only a small number of communities?

• Harder to Conclude that Intervention Caused Change.

Turning a Multiple Baseline Design into a True Randomized Experiment: Roll-Out Design


ROLL-OUT DESIGN • Divide Available Communities into Comparable

Batches • Start Measuring Outcomes for All Communities • Randomly Assign Each Comparable Batch to

WHEN the Implementation Begins • At the end, ALL Communities Are Exposed

• Analysis Uses All Communities and All Times

• Communities Still Serve as Own Controls • Communities Compared by Exposure Status Across Time

Roll-Out Randomized Trials for Dissemination Research (Brown et al.,

2006 2008)


Population

1

2 3 4

5

Time of Transition in Dissemination Randomly Determined

R

Equivalent Subsets that are Ordered

Randomly

Timing of Dissemination ( 0 to x)


• 1 0 0 x x x x x

• 2 0 0 0 x x x x

• 3 0 0 0 0 x x x

• 4 0 0 0 0 0 x x

• 5 0 0 0 0 0 0 x

Random Assignment Reduces Bias in the Long Run


• Example: mPowerment Community Intervention Yr 1 Yr 2 . . . Yr 10

Treat Control Treat Control

R

R

R

Implication of Roll-Out Designs for Community Research


• Units (communities) that get randomized are Large and often few are available at a time

A Single Trial with a Small Number of Communities is

Nearly Always Underpowered • Randomize small numbers of communities now • Randomize small numbers next year • … • Randomize small numbers in following years • Combine results across the years

• Brown et al., Ann Rev Public Health 2009 • Brown et al., Prev Science 2011

Roll-Out Randomized Trials Are Often Acceptable to Communities and

Researchers • Traditional Control Group is Nonstarter for Many

Communities

Why Communities Are Comfortable Participating • Going Early – get a potentially valuable intervention • Going Later – potentially improved intervention • Randomization is Fair

Statistical and Logistical Reasons for Roll-Out Design • Higher statistical power over traditional wait-listed designs • Robust to External Influences • Higher Training Achieved When Training in Small Groups

40 Family Research Consortium San Juan

July 2011

Examples of Roll-Out Trials Now Being Conducted

• 2 Generations of School-Based Suicide Prevention Trials

• Evaluation of the Training of Suicide Prevention Hot Lines

• Randomized Implementation Trial of 40 Counties in California for an Evidence-Based Foster Care Program (MTFC)

Family Research Consortium San Juan July 2011 41

Randomize 40 Counties in CA Chamberlain et al., Adm Pol MH 2008


40 CA Counties

26 Wait LIsted

CDT

Stnd

Wait Listed

13 Wait LIsted

COHORT 1 COHORT 2 COHORT 3

Where These Designs Are Likely to Go

• Implementation Trials of Evidence-Based Programs

• Inclusion of Programs in Health Reform through Research Based Practice Organizations


Summary • Implementation Designs Classified Along How “Real World” How much control over Implementation

Assignment What Levels Are Measured?

• Numerous Opportunities to Improve Evidence

through Design Enhancements – Obtain wider distribution of communities – Strategic use of random assignment – Roll-Out Randomized Trials

• Community Buy-In • Improved Statistical Properties

44

Where are we going in Implementation Research?


Your Role in the Next Stage of Implementation Research: Relying

on What We Now Know


Your Role in the Next Stage of Implementation Research: Your

Ideas in Building This Field


Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation

Documents