S-START Evaluation

Evaluation Team•Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics•Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics•Data Manager: Kyle Kozlovsky, M.S. – Dept Pediatrics•Qualitative Expert: Suyen Schneegans, M.A. – Dept Pediatrics

®

Special thanks to: Rasheem BattleAlejandro BocanegraMeghan CrabtreeMerced DoriaDestiny RamosDrew Russell

Process Evaluation: How is the program being implemented?

S-START Process Goals

1. Train UTHSCSA medical residents and residents from other participating South Texas programs to use evidence based SBIRT procedures for patients who have or are at risk of substance abuse disorders.

2. Promote systems change in targeted residency programs by integrating the SBIRT model into the curriculum on a long-term basis.

Process Goal #1: Train UTHSCSA medical residents and residents from other participating South Texas programs

Key Activities and Measures

1) Develop/implement a comprehensive curriculum

2) Train UTHSCSA and other Faculty Demographics, type of training, satisfaction (GRPA & qualitative

findings)

3)Train UTHSCSA residents & residents from other programs Demographics, type of training, satisfaction (GPRA)

SBIRT Curricular Strategies by Specialty

Pediatrics

OB-GYN

Psychiatry

Internal

Medicine

IM - ERAH

C

FCM

FM –McAlle

n

FM -Fort Hood

FM -Santa Rosa

Surgery

Large Group Didactic Lecture

+ + + + + + + + +

Small Group Discussion

+ + + + + + +

Skill-Building Workshops

+ + + +

Reading Assignments

+ + + +

Screening Questions In Medical Records

+ + + + + + + + + + (likely but not in place

yet)

Reminder Pocket Cards

+ + + +

Screening Assignments*

+ + +

Inpatient Clinical Supervision

+ + + + + +

Outpatient Clinical Supervision

+ + + (child psych only)

+ + + + + +

Independent Study Module (Blackboard)

+ + + +

OSCE (Observed Standardized Clinical Exams)

+ +

What is the core SBIRT event?

Faculty TrainingWho? How

Many?Method/Approach

Pediatrics, UTHSCSA 8 Train-the-trainerFamily & Community Medicine, UTHSCSA

9 Train-the-trainer

Family Medicine, CHRISTUS Santa Rosa

9 Train-the-trainer

Family Medicine, Fort Hood 5 Train-the-trainerInternal Medicine, UTHSCSA 1 Chief Residency Immersion

Training (CRIT)Internal Medicine, ERAHC 1 Train-the-trainer; Chief

Residency Immersion Training (CRIT)

Other UTHSCSA Departments and external Departments

18 Personal consultation, Sharing resources via website, Newsletter, Email

Demographics of Faculty Completing GPRAs

Demographics or Respondents at Baseline

N = 17 (%)

UTHSCSA Family and Community Medicine Pediatrics

8 (47%)

9 (53%) Male

9(52.9%)

Hispanic/Latino 7(41.2%) White

12 (70.6%)

Asian 3(17.6%) African-American 0(0%)

* Wilcoxen Signed Ranks Test

Baseline vs. 30-day Faculty GPRA ratings (N=17)

GPRA Item Mean Baselin

e Rating

Mean 30-Day Rating

Z * P (2- tailed

)

Overall how satisfied with your training experience?

2.12 1.93 -.53 NS

Material presented useful to me in dealing with substance abuse

1.82 1.94 -.51 NS

Training enhanced my skills in topic area 2.00 1.94 .00 NS

Training relevant to my career 1.76 1.88 -.63 NS* Wilcoxon Signed Ranks Test

Qualitative Study of Faculty Perceptions of S-START Purpose: Gain an in-depth understanding of the experience and

perceptions of S-START faculty

Methods

16 training faculty from 5 specialties invited

15 accepted (12 from UTHSCSA; 1 FM program in McAllen, 1 FM CHRISTUS Santa Rosa, 1 FM from Fort Hood)

Mean age (43.93 years); 72% female; 73% MDs, 1 Ph.D., 1 PsyD, 1 M.A.

Average yrs of experience = 15

Qualitative Study of Faculty Perceptions of S-START

Data Collection

Data collection: ≈ 22 months after S-START began

45 to 60 minute interviews using scripted but open-ended questions

14 of the interviews taped to facilitate transcription

Topics: How S-START implemented in the program, barriers and challenges, impact of potential clinical service reimbursement in facilitating program; suggestions for improvement


Data Analysis

Evaluation team hand-coded transcribed interviews

Thematically coded them to correspond to each question

Collapsed materials thematically into 10 emergent or preset categories


Results3 Thematic categories accounted for > 50% of interview

responses Critical components Barriers Motivation

Critical Components Faculty training

Barriers Lack of Leadership

Motivation Buy-in from faculty and residents

Resident TrainingDepartment Trained to

date Expected trained by Year 05 Old estimates

Total Expected Trained by Year 05

Pediatrics - UTHSCSA 80 107 112Family & Community Medicine

•UTHSCSA•McAllen (South Texas)•CHRISTUS Santa Rosa•Fort Hood (Military)

71292517

8424----

95414031

Internal Medicine•UTHSCSA•ERAHC (South Texas)

11316

163--

16326

OB-GYN -UTHSCSA 29 41 41Psychiatry - UTHSCSA 71 120 107Surgery- UTHSCSA 58 --- 96Nurse Practitioners-UTHSCSA

39* --- 39

Total 548 539 679

Demographics of Residents Completing GPRAs

Demographics or Respondents at Baseline

N = 409 (%)

Family Medicine Pediatrics Internal Medicine OB/GYN Psychiatry

138 (33.7%)80 (19.6%)121 (29.6%)

27 (6.6%)43 (10.5%)

Male

155(37.9%)

Hispanic/Latino 133(33.8%) White 259 (63.3%) Asian 81(19.8%) African-American 17(4.2%)

* Wilcoxon Signed Ranks Test

Baseline vs. 30-day Resident GPRA ratings (N=409)

Item Mean Baselin

e Rating

Mean 30-day Rating

Z * P (2- tailed)

Overall how satisfied with your training experience?

1.75 1.71 -4.11 .00

Material presented useful to me in dealing with substance abuse

1.68 1.64 -3.98 .00

Training enhanced my skills in topic area

1.75 1.67 -3.09 .01

Training relevant to my career 1.59 1.50 -2.92 .01

Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SBIRT model into curriculum on long-term basis

Key ActivitiesCouncil of Residency SBIRT Trainers Meetings

Elicit support of key personnel

Changes to Electronic Medical Record

Pocket Cards

SBIRT resources (including key modules) on the S-START website

iPad Project

Progress towards Goal 2:Council of Residency SBIRT Trainers Meetings

Date Pediatrics

BAMC

FCM

Internal Medicine

Psychiatry

OB-Gyn

FM – Fort Hood

FM – Santa Rosa

Trauma

Nursing

1.29.09 2 1 1 - 2 1 - - 1 -1.30.09 2 - 2 2 - - - - - -9.29.09 4 2 1 2 2 - - - 3 -12.08.09

5 1 2 1 1 1 - - - 1

3.09.10 6 1 - - 1 - 1 1 - 18.17.10 4 2 2 2 1 1 - - - -2.15.11 5 - 2 1 - - - - 1 -6.28.11 2 2 - - - 1 - - - -7.19.11 5 - 2 1 1 - - - 3 -

Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SPIRT model into curriculum on long-term basis

Progress re other ActivitiesSupport of change leaders- e.g. UTHSCSA President, Residency

Program Directors; PD and Co-PD have high profile positions

Changes to Electronic Medical Record- UTHSCSA – DFCM, Peds; (Psychiatry and Surgery planned)

Pocket cards McAllen FM: Part of every patient visit paperwork

SBIRT resources (including core modules, resource directory) on the S-START website

iPad Project-proposed for UTHSCSA Pediatrics

S-START Outcome Goals:1. Enhance residents’ knowledge of evidence-based

SBIRT practices.2. Enhance residents’ readiness and perceived

confidence to implement SBIRT with their patients3. Increase residents’ implementation of SBIRT

practices with their patients4. Enhance Faculty Participants’ knowledge and

confidence in ability to teach SBIRT practices to future physicians

What is the program’s impact?

3 x 2 Repeated MeasuresThree data collection methodsMeasurement Occasions for Surveys:

Pre-Test30-Day Follow-UpAnnually up to 36-month follow-up

Measurement Occasions for Pocket CardsVaries by department

Measurement Occasions for Chart Reviews12-month period prior to first core SBIRT module implementation12-month period following the first year of SBIRT module

implementation12-month period following the third year of SBIRT module

implementation

Outcome Design

Evaluation MeasuresDOMAIN

Measure Description Source

Knowledge

Core SBIRT knowledge every resident should know

Local

Residency-specific

Department-specific SBIRT knowledge

Attitudes Readiness to use SBIRT

Readiness to screen patients for use, assess readiness to change, perform intervention/referral, & documentation

Alcohol Education Survey (D’Onofrio et al., 2002)Confidence to

use SBIRTConfidence to screen patients for use, assess readiness to change, perform intervention/referral, & documentation

Current practice

Self-reported current use of SBIRT

Self-reported current practice of screening patients for use, assessing readiness to change, performing intervention/referral, & documentation

Alcohol Education Survey (D’Onofrio et al., 2002)

Pocket cards Family Medicine programs documenting SBIRT behavior w/patients

ASSIST (WHO, 2002)

Chart review Review of charts in Ped. & FCM inpatient clinics to see change in use & documentation of SBIRT

Local

Tool Pre-test

Baseline

30-Day F/U

12- mos F/U

24- mos F/U

36 –mos F/U

GPRA X XAlcohol

Education Survey

X X X X X

Brief Substance Abuse Attitude

Survey

X X X X X

Core knowledge

X X X X X

Residency-specific

knowledge

X X X X X

Incentive $20 $10 $10 $20

Timeline of Self-Administered Instruments & Incentives

Methods of Survey Data CollectionWeb-based surveys (i.e., SurveyMonkey)

Emails to UTHSCSA and private email addressesUnique web links provided to residency

coordinators

Hard copy surveys Pass out at grand rounds and conference periodsIntra-office mail for fellows, facultyMail to home and clinic physical address

Strategies for survey follow-upCollected contact information using a comprehensive tracking

formText reminders to cell phone numbersPhone calls to (1) cell, (2) home, (3) significant others, (4) clinicContact residency coordinators for updated contact informationEnlist authoritative support of facultyLook up information using White Pages, AMA DoctorFinder,

respective state medical board websites (usually Texas)Peer-to-peer contact updates

Future strategies for survey follow-upReminder postcards sent twice before each

annual surveyBring surveys to end-of-year gatherings for

graduating residentsInclude surveys in residents’ exit processing

before graduation I pity the fool who doesn’t take

the survey.

Resident survey ratesPre-

test**Baseline

**30-day 12-

month24-

monthTotal residents & students

153.4% 128.7% 89.9% 56.6% 67.0%

Response rates for similar populations (e.g., students, medical professionals) tend be 60% or lower on follow-up surveys (Asch et al., 1997; Kaplowitz et al., 2004; Kaspryzyk et al., 2001; McMahon et al., 2003; Porter & Whitcomb, 2007)

Analyses of resident survey dataDemographic data (pre-test)Measured changes from pre-test to 12-

month follow-up in:

Selected departments for analysis: Pediatrics, Family and Community Medicine, Internal Medicine

Resident DemographicsPre-test results

TotalN=46

5

UT Ped

n= 97

UT OBn = 29

UT Psy

n = 83

UT Surn = 49a

FMn = 137

IMn = 130

Gender (freq. male)

38.3% 20.6% 10.7% 38.6% - - - 44.9% 51.6%

Race (freq.)

White44.1% 55.2% 67.9% 61.3% - - - 32.6% 31.5%

Black 3.7% 2.1% 3.6% 7.5% - - - 1.5% 4.8%Hispanic 31.1% 28.1% 21.4% 18.8% - - - 35.6% 38.7%

Asian 17.9% 12.4% 7.1% 10.0% - - - 27.4% 19.4%Other 3.2% 2.1% - - - 2.5% - - - 3.0% 5.6%

Age (mean; sd)

30.3(4.8) 28.2(2.6) 28.8(2.4

) 31.4(5.9) - - - 32.5(5.7)

29.1(3.6)Note. UT=University of Texas Health Science Center at San Antonio; Ped.=Pediatrics; FM=Family

Medicine; IM=Internal Medicine; OB=Obstetrics/Gynecology; Psy.=Psychiatry (adult & child); Sur=Surgery.aSurgery residents began the SBIRT curriculum on August 15, 2011.

Resident Demographics (cont.)Pre-test results

UT FCM

n = 72UT McAn = 23

SRn = 24

FHn = 18

UT IMn = 109

UT ERAHCn = 21

Gender (freq. male)

40.0% 47.8% 33.3% 90.9% 48.6% 66.7%

Race (freq.)

White22.5% 9.1% 54.2% 72.2% 35.0% 14.3%

Black 2.8% 0% 0% 0% 1.9% 19.0%Hispanic 28.2% 77.3% 45.8% 0% 36.9% 47.6%

Asian 42.3% 13.6% 0% 22.2% 19.4% 19.0%Other 4.2% 0% 0% 5.6% 6.8% 0%

Age (mean; sd)

33.5(6.2) 35.3(4.6) 28.2(2.4) 30.5(4.4) 28.8(3.5) 30.8(3.9)

Note. UT=University of Texas Health Science Center at San Antonio; FCM=Family and Community Medicine; McA.=McAllen Family Medicine; SR=CHRISTUS Santa Rosa Family Medicine; FH=Fort Hood Family Medicine; IM=Internal Medicine; ERAHC=Edinburgh Regional Academic Health Center Internal Medicine.

Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices.

Core SBIRT knowledge12 items developed locally by the SBIRT project

directorsKnowledge that residents across all departments

should know after training

Residency-specific SBIRT knowledge7-17 items developed locally by the SBIRT faculty in

the respective programsItems designed for specific residency program

SBIRT knowledge and patient populations

Sample core knowledge item:“How many ‘standard drinks’ are considered at-risk alcohol use

by a healthy 40-year-old man?”Sample Pediatrics knowledge item:

“________ exposure is the leading known preventable cause of mental retardation.”

Sample Family and Community Medicine knowledge item:“Hepatitis B, hepatitis C, HIV and AIDS are strongly associated

with abuse of…”Sample Internal Medicine item:

“Alcohol withdrawal treatment on the inpatient medical service is best accomplished by…”

Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices.

Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices.

Core SBIRT knowledgeAll residents increased

SBIRT knowledge, F(1, 167) = 32.1, p < .001.

No differences found between residency programs

Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices.

Residency-specific SBIRT knowledge

Outcome goal 2Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.

Readiness to use SBIRT (D’Onofrio et al., 2002)Subscale of AES comprised of 7 10-point Likert scale itemsRange: 0-100Sample item: “How ready are you to change your practice

behavior to ask patients about quantity and frequency of their alcohol use?”

Confidence to use SBIRT (D’Onofrio et al., 2002)Subscale of AES comprised of 7 5-point Likert scale itemsRange: 0-100 Sample item: “I am confident in my ability to discuss/advise

patients to change their drinking behavior.”

Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.Readiness to use SBIRT: No significant change in

readiness from pre-training to 12 months post-training, F(1, 161) = .87, p = .353.

FCM reported higher readiness than IM overall, F(2, 161) = 4.7, p = ..010.

Pediatrics was not significantly different than the other two programs

Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.Confidence to use SBIRT: Residents overall

reported higher confidence at 12-month, F(1, 161) = 27.3, p < .001.

FCM reported higher confidence overall than IM and Pediatrics, F(2, 161) = 8.1, p < .001.Pediatrics was not

significantly different than the other programs

Outcome goal 3Increase residents’ implementation of SBIRT practices with their patients.Self-report of current SBIRT practice

(D’Onofrio et al., 2002)Subscale of AES comprised of 7 5-point Likert

scale itemsRange: 0-100Sample item: “How often do you formally

screen patients for alcohol problems using brief screening tools (e.g., T-ACE, AUDIT, CAGE)?”

Pocket cardsChart reviews

Current practice of SBIRT skills:

Residents overall reported higher current SBIRT practice at 12-month, F(1, 161) = 35.2, p < .001.

Significant interaction, F(2, 161) = 19.7, p < .001.Both Pediatrics and

FCM improved self-reported current practice .

Internal Medicine declined in self-reported current practice.

Outcome goal 3 cont.Increase residents’ implementation of SBIRT practices with their patients.

Summary of Resident Survey Data FindingsSBIRT core knowledge improved from pre-test to

12-month follow-upReadiness to implement SBIRT did not change,

but was high at pre-testConfidence to use SBIRT improved from pre-test

to 12-month follow-upFor self-report of SBIRT practice, residents

overall improved from pre-test to follow-upHowever, when departments were analyzed

separately, Internal Medicine decreased from pre-test to 12-month

Settings:Family Medicine inpatient service at University

Hospital in San Antonio, Texas

Subjects:285 adult patients, from July 2009 to May 2011.Average Age: 47Gender Distribution: 71.3% Male

Outcome goal 3 cont.UTHSCSA Family Medicine Pocket Cards

Patients were interviewed with a 4-step pocket card Step 1: Pre-screening questions for substance useStep 2: WHO ASSIST (Alcohol, Smoking, and Substance Involvement Screening Test)Step 3: ASSIST score to assess the level of risk and determine need for interventionStep 4: checklist describing the intervention, patient response, and future plan.

Residents were asked to complete 12 per year26 out of 26 trained residents participatedResidents completed 11 total on average

UTHSCSA Family Medicine Procedures

Step 1: Pre-screening Results

95.8% of patients screened positive for at least 1 substanceAvg. ASSIST Score was 19 indicating a moderate risk of substance abuse

Step 2-3: ASSIST Results

When the ASSIST Score recommended a brief intervention, residents reported some form of brief intervention 69.4%(over two thirds ) of the time

Residents most likely to discuss consequences of use if ASSIST Score recommended brief intervention (79% of the time)

8% of patients declined to discuss their response to screening

Brief Interventions

Brief Intervention Actions Taken

When the ASSIST Score suggests a referral to treatment, residents referred a patient to treatment 71.8% of the time

Residents were most likely to contact an LCDC (Licensed chemical dependency counselor) when ASSIST Score recommended a referral to treatment (46.5% of the time)

Referrals to Treatment

Referrals to Tx

Referrals to Treatment Actions Taken

Patients were more likely to report some sort of action (cut back, quit or seek outside help) when the resident documented a brief intervention (80% of the time compared to 71.4%)

Patients were more likely to report some sort of action when the resident documented a referral to treatment (80% of the time compared to 68.8%)

Step 4: Patient Plans

The ASSIST Pocket Card was distributed to other departments with some participation

Participation in Other Departments

Residency # of Cards Completed

# of Residents

Participating

# of Residents Trained

UTHSCSA McAllen Family Medicine

103 13 29

Fort Hood Family Medicine

25 6 11

Outcome goal 3 cont.Overall chart review study design

Location 12-month period prior to first core SBIRT module implementation

12-month period following the first year of SBIRT module implementation

12-month period following the third year of SBIRT module implementation

Pediatrics May 4, 2008-May 4, 2009

May 4, 2010-May 4, 2011

May 4, 2012-May 4, 2013

Family & Community Medicine

April 22, 2008-April 22, 2009

April 22, 2010-April 22, 2011

April 22, 2012-April 22, 2013

Goal 3 cont.Overall Chart Review Study Design

1st Chart Review*

# of cases for

sample1. Pediatrics outpatient continuity clinic 4001. Family & Community Medicine outpatient

continuity clinic 400SUBTOTAL 800

2nd Chart Review^

# of cases for



3rd Chart Review

# of cases for



TOTAL 2400

*Completed^In Progress

Setting: Pediatrics Outpatient Continuity Clinic

Chart selection criteria: (1) seen at least once within the review period by a

resident(2) at least one visit within the appropriate review

period (e.g., one year prior to implementation of core SBIRT intervention and medical record changes, one year after, and three years after)

(3) age of patient 11 and upVisits included in the analysis are acute visits and annual

well child visits

Pediatrics Chart ReviewBaseline findings

Patient DemographicsMean age was 13

(1.98)

49.5% Female

Ethnicity known to be majority Hispanic, but documentation in charts is rare

ScreeningOut of 967 visits, 511 (just over half) screenings

were documentedVisits included Annual Check ups and Acute Visits

Some acute visits were sports physicals

Screening Cont.9 positive screenings documented for tobacco0 positive screenings documented for alcohol

or other drugsThe HEADSS was documented as a

screening tool in 149 visitsOnly one CRAFFT

screening was documented

Brief Interventions and Referrals to TreatmentOf 967 visits, 204 brief interventions were

documented4 of 9 positive tobacco screenings were

followed by a documented brief intervention“Anticipatory guidance” for drug use is often

used with pediatric patients and was considered a BI in our design

Only 2 referrals to treatment were documented

Conclusions from Pediatric Chart ReviewDocumentation is a likely contributor to the

lack of SBIRT practices foundOngoing changes to the medical record and

routine SBIRT training will likely increase SBIRT practices

An increase in screening might lead to an increase in positive screenings (and, in turn, increase opportunities for brief interventions and referrals to treatment)

Outcome goal 4Enhance faculty participants’ SBIRT knowledge and confidence in ability to teach SBIRT practices to future physicians. SBIRT knowledge

Developed locally by the S-START project directorsSame core knowledge that the residents receive

Confidence to teach SBIRT to residents Adaptation to the resident scale (D’Onofrio et al.,

2002)Range: 0-100Sample item: “I am confident in my ability to train

residents in advising patients to change drinking behavior. “

Outcome goal 4 cont.Enhance faculty participants’ SBIRT knowledge and confidence in ability to teach SBIRT practices to future physicians.

S-START Evaluation

Documents