Developing a progressive approach to using the GAIN in order to reduce the duration and cost of assessment with the GAIN short screener, Quick, and Computer.

Developing a progressive approach to using the GAIN in order to reduce the duration and cost of assessment with the GAIN short screener, Quick, and Computer Adaptive Testing

Michael L. Dennis, Rodney R. Funk, Janet C. Titus, Barth B. Riley, Chestnut Health Systems, Bloomington, IL

Sean Hosman, Sarah Kime, Assessments.com, Salt Lake City, UTPanel at the Joint Meeting on Adolescent Treatment Effectiveness, March 25-27, 2008, Washington, DC. This presentation supported by Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) contracts 270-2003-00006 and 270-07-0191, as well as several individual CSAT, NIAAA, NIDA and private foundation grants. The opinions are those of the author and do not reflect official positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827-6026, fax: (309) 829-4661, e-Mail: [email protected]

Background The Global Appraisal of Individual Needs (GAIN) is a family of

instruments ranging from screeners to full standardized biopsychosocial.

Over the past decade it has become one of the most commonly used standardized instruments in adolescent substance abuse treatment and research.

Its strengths include- mapping onto common standards for diagnosis, treatment, planning, placement,

and performance monitoring; - clear training, feedback and certification standards to support workforce

development and interpretation; - a large number of investigators conducting primary studies, methodological and

secondary analyses with it; - a large pooled data facilitating comparisons across studies, sites and evidenced

based practices; and the introduction (in 2008) of a new web based infrastructure to support use across multiple settings.

Purpose of this panel The biggest single complaint about the GAIN is the time it

takes to administer (90-120 minutes for a full GAIN) and the cost (both in terms of training and staff time to administer).

This provides an overview of the progressive approach to assessment we are trying to develop and then talk about three specific efforts we are working on to reduce the duration and cost of assessment.

1. Introduction of the GAIN Short Screener (GSS)

2. Developing a newer and more efficient version of the GAIN –Quick (GQ)

3. Developing a computer adaptive testing (CAT) version of the full GAIN

Progressive Assessment Approach Screening to Identify Who Needs to Be Fully “Assessed”

- Focus on brevity, simplicity for administration & scoring- Needs to be adequate for triage and referral

Quick Assessment for Targeted Referral - Assessment of who needs a feedback, brief intervention or referral for more

specialized assessment or treatment- Needs to be adequate for brief intervention

Comprehensive Biopsychosocial - Used to identify common problems and how they are interrelated- Needs to be adequate for diagnosis, treatment planning and placement of

common problems Specialized Assessment

- The bio-psycho-social may identify areas where additional assessment by a specialist (e.g., psychiatrist, school counselor) may be needed to rule out a diagnosis or develop a treatment plan or individual education plan

Program Level Assessment- For program management, evaluation, and planning

Reducing assessment time and cost by using the GAIN shortscreener

Michael L. Dennis, Rodney R. Funk,Chestnut Health Systems, Bloomington, IL

Sean Hosman, Sarah Kime, Assessments.com, Salt Lake City, UT

Overview of the GAIN-SS A 3- to 5-minute screener Used in general populations to identify or rule-out clients who

will be identified as having a behavioral health disorders on the 60-120 min versions of the GAIN

Easy for use by staff with minimal training or direct supervision

Provides a measure of change Designed for self- or staff-administration, with paper and pen,

computer, or on the web Translated by collaborators into several languages including

French, Japanese, Portuguese, and Spanish so far

Factor Structure of GAIN Measures of Psychopathology

Source: Dennis, Chan, and Funk (2006)

Recency Response Set Recency of 20 problems rated as past month (3), 2-12

months ago (2), more than a year ago (1), never (0) Combined by cumulative time period as:

- Past month count (3s) to measure of change- Past year count (2s or 3s) to predict diagnosis- Lifetime count (1s, 2s or 3s) as a measure of peak severity.

Can be classified within time period low (0), moderate (1-2) or high (3)

Can also be used to classify remission as - Early (lifetime but not past month)- Sustained (lifetime but not past year)

Item Selection Using a Rasch Measurement Model

-1.89 -.81 -.32 +.28 +.71

Items around key decision point

Internalizing Disorder Screening (IDScr)

Externalizing Disorder Screening (EDScr)

Substance Disorder Screening (SDScr)

Crime/violence Disorder Screening (CVScr)

Some important caveats… Does not focus on low level problems (e.g., any use)

to minimize false positives Designed to have 90% sensitivity and is only self

report, which means it does miss some and that clinicians should still be able to over ride on a case by case basis

Inclusion of lifetime measures provide checks against temporary phenomena (e.g., being in a controlled environment) and to drift when used to measure change (e.g., lifetime use should never go down)

Just a screener, not a full assessment

Example of GAIN SS Triage Profile

13

5

3 3

2

0

4

8

12

16

20

Total Disorder Screener (TDScr)

InternalizingDisorder Screener(IDScr)

ExternalizingDisorder Screener(EDScr)

SubstanceDisorder Screener(SDScr)

Crime/Violence Screener(CVScr)

0

1

2

3

4

512+ mon.s ago (#1s)2-12 Mon.s ago (#2s)Past Month (#3s)Lifetime (#1,2,or 3)

L

ow

Med

ium

H

igh

1a-4e1a-1e 2a-2e 3a-3e 4a-4e

L

ow

M

ediu

m

Hig

h

L

M

Hig

h

Example of Monitoring with GAIN SS

109

11

910

8

32 2

0

4

8

12

16

20

Intake 3Mon

6Mon

9Mon

12Mon

15Mon

18Mon

21Mon

24Mon

Total Disorder Screener (TDScr)

12+ mon.s ago (#1s)

2-12 Mon.s ago (#2s)

Past Month (#3s)

Lifetime (#1,2,or 3)

Track Gap Between Prior and current

Lifetime Problems to identify “under

reporting”

Track progress in reducing current

(past month) symptoms)

Monitor for Relapse

Using GAIN SS to Help with Placement Decisions for Adolescents Substance Users

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

11%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

TDScr Score

% w

ithi

n L

evel

of

Car

e an

d A

ge G

roup OP/IOP (n=2499)

Residential (n=1965)

Low

Mod High ->OP/IOP

Median=6.0Residential

Median=10.5

Psychometric Properties

Total Disorder Screener (TDScr)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Prevalence (% 1+ disorder)

Sensitivity (% w disorder above)

Specificity (% w/o disorder below)

(n=6194 adolescents)

Low Mod. High

99% prevalence, 91% sensitivity, & 89%

specificity at 3 or more symptoms

Using a higher cut point increases prevalence and specificity, but

decreases sensitivity

Total score has alpha of .85 and is

correlated .94 with full GAIN version

Source: Dennis et al 2006

GSS Performance by Subscale and Disorders

Prevalence Sensitivity Specificity Screener/Disorder 1+ 3+ 1+ 3+ 1+ 3+ Internal Disorder Screener (0-5) Any Internal Disorder 81% 99% 94% 55% 71% 99% Major Depression 56% 87% 98% 72% 54% 94% Generalized Anxiety 32% 56% 100% 83% 44% 83% Suicide Ideation 24% 43% 100% 84% 41% 79% Mod/High Traumatic Stress 60% 82% 94% 60% 55% 90%

External Disorder Screener (0-5) Any External Disorder 88% 97% 98% 67% 75% 96% AD, HD or Both 65% 82% 99% 78% 51% 85% Conduct Disorder 78% 91% 98% 70% 62% 90%

Substance Use Disorder Screener (0-5) Any Substance Disorder 96% 100% 96% 68% 73% 100% Dependence 65% 87% 100% 91% 30% 82% Abuse 30% 13% 89% 25% 14% 28%

Crime Violence Screener (0-5) Any Crime/Violence 88% 99% 94% 49% 76% 99% High Physical Conflict 31% 46% 100% 70% 38% 77% Mod/High General Crime 85% 100% 94% 51% 71% 100%

Total Disorder Screener (0-5)Any Disorder 97% 99% 99% 91% 47% 89% Any Internal Disorder 58% 63% 100% 98% 8% 28% Any External Disorder 68% 75% 100% 99% 10% 37% Any Substance Disorder 89% 92% 99% 92% 20% 51% Any Crime/Violence 68% 73% 100% 96% 10% 32%

Low (0), Moderate (1-2), and High (3+) cut points can

be used to identify the need

for specific types of

interventions

Moderate can be targeted where resources allow or where a more

assertive approach is

desired

Mod/Hi can be used to evaluate

program delivery/referral

Rapidly Spreading State or Provincial wide implementation in multiple states (ID,

CT, LA, MI, MN, NH, NV, OR, SC, WA, WI) and provinces (BC, ON, QU) in one or more large systems (adolescent or adult addiction treatment, mental health, welfare, juvenile or criminal justice , Student or Employee Assistance Programs),

Used by SAP or EAP in Brazil, Canada, Japan, Mexico, United States and being translated for use in China.

Most have relied on paper and pencil administration or installed into their own information system – but then do not get reports.

There is a software template available, but its use still requires adaptation to one’s own system

Assessments.com is one of the one of the first commercial vendors to offer a web based version of the GAIN SS for both system and individual level user.

References Dennis, M.L., Chan, Y-.F., & Funk, R.R. (2006). Development and validation of the GAIN Short Screener

(GAIN-SS) for psychopathology and crime/violence among adolescents and adults. The American Journal on Addictions, 15(supplement 1), 80-91. Available from http://www.chestnut.org/LI/gain/GAIN_SS/Dennis_et_al_2006_Development_and_validation_of_the_GAIN_Short_Screener.pdf .

Dennis, M. L., Feeney, T., Stevens, L. H., & Bedoya, L. (2007). Global Appraisal of Individual Needs–Short Screener (GAIN-SS): Administration and Scoring Manual for the GAINSS Version 2.0.1. Bloomington, IL: Chestnut Health Systems. Retrieved on from http://www.chestnut.org/LI/gain/GAIN_SS/index.html .

AcknowledgementsThe above manual, articles and this presentation were supported by the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) under contracts 207-98-7047, 277-00-6500, 270-2003-00006, and 270-07-0191 using data provided by the following grantees: CSAT (TI-11320, TI-11324, TI-11317, TI-11321, TI-11323, TI-11874, TI-11424, TI-11894, TI-11871, TI-11433, TI-11423, TI-11432, TI-11422, TI-11892, TI-11888, TI013313, TI013309, TI013344, TI013354, TI013356, TI013305, TI013340, TI-130022, TI03345, TI012208, TI013323, TI-14376, TI-14261, TI-14189,TI-14252, TI-14315, TI-14283, TI-14267, TI-14188, TI-14103, TI-14272, TI-14090, TI-14271, TI-14355, TI-14196, TI-14214, TI-14254, TI-14311, TI-15678, TI-15670, TI-15486, TI-15511, TI-15433, TI-15479, TI-15682, TI-15483,TI-15674, TI-15467, TI-15686, TI-15481, TI-15461, TI-15475, TI-15413, TI-15562, TI-15514, TI-15672, TI-15478, TI-15447, TI-15545, TI-15671, TI-11320, TI-12541, TI00567); NIAAA (R01 AA 10368); NIDA (R37 DA11323, R01 DA 018183); the Illinois Criminal Justice Information Authority (95-DB-VX-0017); the Illinois Office of Alcoholism and Substance Abuse (PI 00567); the Intervention Foundation’s Drug Outcome Monitoring Study (DOMS); and the Robert Woods Johnson Foundation’s Reclaiming Futures project. Any opinions about these data are those of the authors and do not reflect official positions of the government or individual grantees. Thanks to Janet C. Titus, Joan I. Unsicker, Rod Funk, Ya-Fen Chan, Michelle White, Lexy Adkins, Tim Feeney and David Smith for their help in writing this manual. Thanks to Sandra McGuinness for developing the software application. Suggestions, comments, and questions can be sent to Dr. Michael Dennis, Chestnut Health Systems, 720 West Chestnut, Bloomington, IL 61701, [email protected] .