1 From Efficacy to Effectiveness: the Safe in the City Model 2009 Meeting of the International Society for STD Research June 28 – July 1, 2009 London 2009.

1

From Efficacy to Effectiveness:the Safe in the City Model

2009 Meeting of the International Society for STD Research June 28 – July 1, 2009

London

2009 Meeting of the International Society for STD Research June 28 – July 1, 2009

London

2

Efficacy vs. Effectiveness

• Efficacy: How well an intervention works in optimal settings– Well-trained and paid research staff– Controlled Setting– Carefully recruited study subjects– Carefully constructed and executed

research protocol

3

Efficacy vs. Effectiveness

• Effectiveness: How well an intervention works in the real world– Staff not specifically trained for intervention; and

need to meet competing demands– Targets all patients/clients, not selected subjects– Adaptation of intervention protocol often

necessary to make it practical for the setting (threat to fidelity)

4

Effectiveness Research

• Implicitly takes into account the acceptability, feasibility, implementation, and impact of an intervention in the settings for which the intervention was designed

5

Effectiveness Research3 Main Components

• Participatory research model

• Efficacy/Effectiveness evaluation

• Scale-up process and evaluation

6

Participatory Research Model

• Involves all stake holders that determine the ultimate success of the intervention:– Academics / intervention developers– Target population representatives– Representatives from the organizations

responsible for implementation

7

Efficacy/Effectiveness Evaluation

• Evaluation of efficacy/effectiveness should take place in an environment that closely mimics the environment for which the intervention is ultimately intended

8

Scale – Up Process and evaluation

• Marketing strategies and implementation

• Evaluation of intervention up-take by target audience

• Short to long term assessment of implementation and sustainability

9

10

Safe in the City Project Investigators

Centers for Disease Control and PreventionLee Warner, Andrew Margolis, Jocelyn Patterson, Craig Borkowf

Denver Public Health Cornelis Rietmeijer, John Douglas, Doug Richardson

Education Development Center, Inc.Lydia O’Donnell, Athi Myint-U, Carl O’Donnell

Long Beach, California State University and Department of Health and Human ServicesKevin Malotte, Shelley Vrungos, Nettie DeAugustine

San Francisco Department of Public HealthJeffrey Klausner, Gregory Greenwood, Carolyn Hunt

11

Safe in the City: Focus on Effectiveness

• Collaborative, participatory research process involving all stakeholders

• Formative process• Intervention outcome study that

involved entire clinic populations• Use of STI markers of effectiveness• Closely-guided and evaluated post-

study dissemination phase

12

Safe in the CityDevelopment of the

InterventionCornelis A. Rietmeijer, MD, PhD

Denver Public Health DepartmentDenver, Colorado, USA

13

Study Rationale

340,000,000 incident STDs worldwide annually

STD clinics provide access to men and women likely to be infected and to acquire new infections over time

Yet behavioral interventions with counseling or multiple sessions are difficult to implement in busy medical settings

Recent interest in simple, easy to use, and low cost interventions for waiting rooms

14

Rationale continued

Previous research suggests benefits of video-based approaches, but subject to limitations:

Controlled research settings Tailored videos Single site Inclusion of group counseling

Effectiveness of stand-alone video in ‘real-world’ setting is unknown

15

Safe in the City Project Overview

5-year CDC-funded multi-site study

Develop a brief video-based STD clinic waiting room intervention to reduce (or eliminate) STI and risky sexual behavior

Evaluate effectiveness in 3 publicly funded STD clinics in Denver, San Francisco, and Long Beach, CA.

16

Intervention Development Considerations

Waiting rooms in medical settings provide an underused opportunity to reach patients who are thinking about their health.

Yet to be feasible and sustainable, interventions must:

Be easy and inexpensive to administer

Result in minimal interruption of patient flow

Require few clinic resources, especially staff time

Be acceptable to diverse clients

17

Formative Process

Identification of intervention medium, theoretical framework, and key messages by research team

Collaboration with award-winning film maker to integrate framework in an appealing product

Multi-step participatory process involving target audience, clinic staff, and community advisors

Intervention research study in 3 STD clinics

18

Intervention Development: Integrated Theoretical

Framework

Core constructs grouped into interconnected elements

→ HIV/STD risk, knowledge, perception→ Positive attitudes toward condom use

→ Self-efficacy/skills for condom negotiation, acquisition, use→ Modeling of appropriate behaviors

Theory of Planned Behavior

Social Cognitive Theory

Information Motivation

Behavior Model

19

Focus Groups

3 sites held 12 focus groups with 176 participants

3 different stages of video development:

Story line development Script development Post-production editing

20

What Is the Intervention?

23-minute video

3 story lines

2 cartoon animations Condom variety and

selection Instructions for use

Posters in waiting and exam rooms

21

Story Line – Paul and Jasmine

Things are getting more serious between Paul and Jasmine, but Paul “slips” and has a sexual encounter with Teresa.

Teresa gets an STD and tells Paul. Now Paul has to tell Jasmine.

22

Story Line – Rubén, Tim and Christina

Rubén’s girlfriend Christina doesn’t know about his interest in men. Rubén and Tim have a casual sex encounter after

meeting in a bar. Days later, Christina suspects something is wrong.

She insists on a visit to the STD clinic.

23

Story Line – Teresa and Luis

Teresa has recently met Luis. After her STD scare with Paul, Teresa is serious about wanting to use

condoms. Now she has to convince Luis.

24

25

26

Assessing Intervention Effectiveness in an STD Clinic Population:

the Safe in the City Model

Lee Warner, PhD

Centers for Disease Control and PreventionAtlanta, Georgia, USA

27

Examples of Challenges in Study Design

• Examining “real-world” effectiveness prohibits active patient enrollment *still need to deliver intervention, include entire clinic population

• Large sample size required, given effectiveness of brief intervention likely modest

• Biologic markers (STI) preferred effectiveness measure to self-reported behavior

• Unable to randomize in waiting room setting, but need balance between study conditions

28

Maximizing Intervention Delivery and Exposure

Identify environmental characteristics of waiting rooms

Observe waiting room flow

Determine appropriate playback frequency

Identify factors to increase viewership (goal: 80%)

Assess and adjust to clinic staff acceptance of video

29

Denver Waiting Room

2nd TV2nd TV

30

Long Beach Waiting Room

31

San Francisco Waiting Room

32

Video Viewership

01020304050

60708090

100

Intervention Cycles

% V

iew

ersh

ip

Viewership Goal

Viewership as defined by watched most or all of the video + identified a main message

33

Study Objective

• To determine whether this brief, structural intervention reduced incident laboratory-confirmed infection among typical visitors to an STD clinic

34

Overview of Study Design Population: = all patients attending 3 STD clinics from

December 2003 – August 2005 (N = ~40,000)

Study design: 2 arm non-randomized controlled trial

Arm assignment: alternating 4-week control & intervention periods

Data collection: retrospective review of clinic data & external surveillance records to ascertain new STI diagnoses, conducted under waiver of informed consent

35

Study Flow Diagram

Patient Presents at ClinicN=38,635

Intervention:23 Minute Video Shown in Clinic Waiting Room

Control: Standard Waiting Room

Experience

Clinical Evaluation Clinical Evaluation

Review of Medical Records & Surveillance Registry Data for STD

outcomes

Review of Medical Records & Surveillance Registry Data for STD

outcomes

Assignment Based on Clinic Visit Date

36

Characteristics of Clinic Populations, by Condition Control

(n = 19,562) Intervention (n = 19,073)

Sex

Male 70% 69% Female 30% 31%

Race/ethnicity

Black, non-Hispanic 19% 18% White, non-Hispanic 46% 46% Hispanic

Other

25%

10%

25% 11%

Age <25

>25 31%

69%

31%

69%

Sexual orientation: MSM

STI at index visit

22%

15%

21%

16%

37

Approach to Statistical Analysis

Outcome: Incident laboratory-confirmed infection – gonorrhea, chlamydia, trichomoniasis, syphilis, and HIV

Analysis: Survival analyses estimating time to infection– Kaplan-Meier analyses comparing survival at specific

points in time by condition*

– Cox proportional hazards regression comparing hazard ratios by condition*

38

Control

N (%)

Intervention

N (%) GC 539 (2.8) 409 (2.1)

CT 666 (3.4) 573 (3.0)

Trich 71 (0.4) 71 (0.4)

Syphilis 40 (0.2) 35 (0.2)

HIV 10 (0.0) 4 (0.0)

Any infection 1,113 (5.7) 929 (4.9)

First New Laboratory-Confirmed Infection,

By Condition

39

Overall Effect of Intervention on Laboratory-Confirmed Infection*

Hazard Ratio (95% CI)

All patients 0.91 (0.84-0.99)

* = 9% reduction in STI incidence* = 9% reduction in STI incidence

40

Intervention Effect, by Characteristic

Hazard Ratio (95% CI)STI at index visit

Yes 0.86 (0.75-0.99)

No 0.93 (0.84-1.04)

Sex

Males 0.87 (0.78-0.96)

Females

Sexual orientation

Heterosexual

MSM

Age

<25

>25

1.06 (0.89-1.25)

0.84 (0.71-0.98)

0.90 (0.79-1.03)

1.02 (0.88-1.17)

0.85 (0.77-0.95)

41

Control Intervention

STD clinic 76.5% 77.3%

Other facilities 23.5% 22.7%

Source of STI Diagnosis, By Condition

p = 0.67p = 0.67

42

Control Intervention

Any visit 40.0% 40.1%

Mean no. visits 0.85 0.80

Return Visits to Clinic, By Condition

p = 0.86p = 0.39p = 0.86p = 0.39

43

Factors Significantly Associated with

Incident STI, by Strength of Effect Measure

• MSM

• Baseline STI diagnosis

• Minority race / ethnicity

• Age < 25 yrs

HighestHighest

LowestLowest

44

Meeting Challenges in Study Design

• Evaluation of intervention effectiveness in “real-world” setting with “passive enrollment” of patient population

• Efficient examination of laboratory-confirmed STI using existing medical / surveillance records

• Inclusion of entire patient population, ensuring generalizability

• Non-randomized design with systematic allocation balanced all measured characteristics

45

Factors Critical to Success of Evaluation

• Ability to examine study population in advance: – Patient flow (e.g., number of patients, waiting room

time)– Reaction of patient / staff to video– STI prevalence and incidence of population known

• Access to electronic medical records and surveillance registries to identify incident STI

• Waiver of informed consent from IRB critical

46

Conclusions “Safe in the City” associated with reduction in incident STI *** important -- estimate = effectiveness, not efficacy.

This reduction can have significant public health benefit, but requires sufficient availability of intervention and wide adoption by clinics.

Relatively* easy-to-implement, low cost interventions can reach large numbers of high-risk populations with minimal effort.

Type of design used to examine effectiveness of “Safe in City” promising for studies of STD prevention interventions.

47

How to Use a Condom,

Christina & Ruben

48

49

50

Safe in the CityDiffusion of the Intervention in the

U.S.

Doug Richardson, MAS

Denver Public Health DepartmentDenver, Colorado, USA

51

Primary study finding publishedPLoS Medicine June 2008

52

It’s alive! Now what?

53

CDC’s Diffusion Goals

Get the word out Make it available Provide support

Coordinated by the Diffusion of Effective Behavioral Interventions (DEBI) CDC & Academy of Education Development Expertise in disseminating STD/HIV trainings and

interventions www.EffectiveInterventions.org

54

CDC/DEBI’s Action Plan

SITC homepage and order form launched on DEBI website the same day the PLoS article is released.

SafeInTheCity.org goes live – view and download the video.

CDC issues a “Dear Colleague” letter encouraging STD clinics to consider implementing SitC.

Podcasts and CDC press release

55

DEBI web site home page

56

Safe in the City – Kit Request Form

57

Safe in the City Kit Materials DVD with video

and pdf posters User’s Guide Now Showing

poster CDC documents

58

Safe in the City Kit Requests

June 24, 2008 – May 28, 2009Agency type # of kit requests

STD Clinics 444

Health Departments 266

Community-Based Organizations 266

Health Services Clinics 258

Family Planning Clinics 190

Others (e.g., Hospitals, Univ. Research, Univ.

Student Health Centers) 421

Total 1,845

60

Follow-up on the Initial Launch

• Call clinics to do a brief survey and to offer technical assistance.– Clinics requesting technical assistance– Random sample STD clinics

• Two net meetings with intervention developers

61

Survey Details

• 128 clinics sampled in Nov 200852 clinics who indicated a possible need for

tech assistance

76 additional randomly selected STD clinics

• 87 completed a baseline survey

• 81 completed a follow-up survey 4-5 months later

62

Site characteristics

Clinic Type (self reported) n %

STD 70 80Health Services 13 15

Family Planning 2 2

Other 2 2Median Patient Annual Volume 2500

63

How did you learn about SitC?

Colleague/word of mouth 26

CDC website 16

CDC’s Dear Colleague Letter 6

EffectiveInterventions.org 5

SafeInTheCity.org 2

STDPreventionOnline.org 2

STD Conference/Meeting 2

PLoS article 0

Don’t know 11

Other 32

64

Sites showing the video

• Baseline – 35 of 87 sites, 40%Mean Days From Video Receipt to Baseline Survey – 117 days

• Follow-up – 46 of 81 sites, 57%Mean Days From Video Receipt to Follow-up survey – 250 days

65

Among sites with follow-up data (n=81)

• Of the 34 sites showing the video at baseline 27 (79%) were showing it at follow-up.

• Of the 47 sites NOT showing it at baseline, 19 (40%) were showing at the follow-up.

66

Reasons for not showing the video (Baseline)

• 33% TV/DVD player purchase

• 15% Not appropriate for their clients

• 12% Pending approval

• 40% Other reasons– Too busy, staffing, lost DVD, using it outside of the

clinic, developing augmenting materials

67

Factors Critical to Successful Diffusion

• Creating a coordinated diffusion action plan

• Proactively assess intervention uptake

• Identify barriers to implementation

68

Additional Resources

• Order the video and view past informative net meetings on the Safe in the City webpage at www.effectiveinterventions.org

• Project website (includes video links) www.safeinthecity.org

69

Thank You!

Questions?