1 Client Level Data Collection: The California Experience Jeffrey Bernstein, MS Center for AIDS Prevention Studies, California Technical Assistance Program.

1

Client Level Data Collection: The California Experience

Jeffrey Bernstein, MSCenter for AIDS Prevention Studies, California Technical Assistance

Program

Shanna Livermore, BS

Steven R. Truax, PhDCalifornia Department of Health Services, Office of AIDS,

HIV Education and Prevention Branch, Sacramento

Presentation for the 4th Annual Center for AIDS Prevention Studies Conference

San Francisco, April 16, 2004

2

Today’s Presentation

• Overview of ELI

• What challenges have been experienced at the state and local levels?

• How does ELI work successfully in California at the state and local levels?

• How has this technology been used to inform public health decisions?

3

ELI Background Overview

Development

4

What is ELI?

• Evaluating Local Interventions (ELI) is a web-based evaluation system that enables California’s prevention providers to collect and access information critical for the planning and implementation of effective HIV prevention programs.

5

Background

• Evaluation is important for planning and enhancing programs and for securing resources.

• Office of AIDS (OA) has long recognized the need for evaluation and technical resources required to support it.

6

Background

• CDC Guidance on evaluation released in 1999 added additional motivation.

• CDC’s and OA’s shift in new directions

7

Shift happens!!

8

Benefits• Systematic data collection

– Gives the client a voice

• Consistent reporting requirements

9

Benefits

• There is a strong relationship between data collection and program planning

• Helps in grantwriting

• Helps structure your services/conversations

10

Benefits

• Immediate feedback to providers– Over 50 “cannned reports”– Raw data available

11

Prevention Activities Required to be Evaluated

• Individual-level interventions

• Group-level interventions

• Outreach interventions

• Prevention case management

• Health Communication

12

ELI Development Partners

• Coordination & Development– California State Office of AIDS

– Universitywide AIDS Research Program/UCOP

• Program Evaluation Technical Assistance– Center for AIDS Prevention Studies, UCSF

• Guidance & Input– Local Health Departments

– CDC

13

Strategic Planning Process for Development of ELI

• Sought input from stakeholders

• Assessed

– existing evaluation efforts

– state of the art measures, methods

– local feasibility and utility

14

Pilot Phase

Fall 2001

15

Pilot Process for ELI

• ELI piloted in 5 counties• Feedback collected on

– data collection forms & protocols

– web system

– resource issues

– technical assistance needs

• Trainings and TA developed• ELI revised

16

Pilot Feedback: Key Concerns• Use of individual-level forms in outreach and

group settings• Need for additional fields, data elements, &

reports• Ability to download data• Integration of existing data collection efforts into

ELI system• Need for training and technical assistance on data

collection, system use, and use of data• Resources

17

OA Response to Pilot Feedback• Revised protocol for data collection in outreach and group

settings

• Created additional data fields, optional fields & reports

• Developed downloading capability

• Designed & implemented statewide trainings

• Developed partnership with CAPS to provide evaluation TA to support use of forms & data

• OA is available for – system technical assistance – consultation for integration of ELI forms into existing data

collection efforts

18

Explanation of forms

19

Prevention Interventions to be Evaluated Using ELI

• Intervention: A specific activity intended to bring about HIV risk reduction– 1 Individual-Level Interventions (ILI)

– 2 Group-Level Interventions (GLI)

– 3 Outreach

– 1 Health Communication

– 1 Prevention Case Management (PCM)

20

Individual Level Interventions

21

Group Check Sheet (front)

22

Group Check Sheet (back)

23

Overview of ELI Elements

• Agency Information

• Intervention Info.–Intervention setting–Intervention type–Referrals–Risk reduction materials

• Demographics–Gender–Race/ethnicity–Sexual orientation–Matching criteria

• Behavioral Info.– Sexual Behavior– Hepatitis & STD History– Substance Use History

• Supplemental Info.

24

What are the Matching Criteria & Why are They Important?

• Date of Birth

• First Letter of Last Name

• Gender

• Race/Ethnicity

• Zip code

25

Selecting Forms

• Discuss form selection

• Forms should be decided on by the time intervention is set-up

• Some degree of form flexibility upon Office of AIDS approval

26

ELI Data Collection Forms

• Not a script!– Provide a framework for a conversation with a

client– Optional word use and question order– Use terms familiar to and appropriate for the

client

27

Explanation of Reports

• Over 50 “canned” ELI reports available– Age by Race by Gender– Missing Fields– Intervention Effectiveness– Referrals by Risk

Raw data available

28

Reports

29

Reports

• Budget Report• Referrals by Risk• Gender by Race by Age Group• Sexual Orientation by Race by Age Group• STD by Race by Gender• STD Type by Race by Gender• Drugs by Race by Age Group• Materials Distributed by Risk• Missing Fields• Intervention List

30

Reports

31

Reports

• Garbage in – garbage out

• Reports are as rich as the data you enter

32

Challenges

33

Challenges

• Time

• Staffing

• Changing CDC requirements

• Funding – Program cuts

34

Challenges

• Individual client data requirements in certain settings

• Federal vs. State vs. Local requirements • Information Technology is not always a

super highway• Different levels of knowledge• Requires new scope of work format• Intervention set-up: LHJ vs. CBO

35

Challenges

• Data Collection– New forms/form integration– Matching Criteria

• BIG challenge with some populations (e.g., clients with immigration issues)

36

Successes

37

Successes

• Increased communication

• Increased collaboration

• Increased technical assistance

• Increased training opportunities

38

Successes

• Accurate timely information about HIV prevention interventions

• Efficiency increase scope of work review

• Intervention Working Group (IWG)

39

Increased focus on quality interventions!

NOT quantity.

40

Characteristics of Successful Interventions

• Science based

• Highly targeted

• Clearly defined goals and objectives

• Realistic goals, objectives and implementation timeline

• Focused on reducing specific risk behaviors

• Adequately trained staff

41

Characteristics of Successful Interventions

• Sensitive to target population

• Sustainable resources and support

• Flexible

• Part of community plan

• Culturally, developmentally competent

• Gender specific

42

Lessons Learned

• Secure funding ahead of time

• Lengthen and expand the pilot phase

• Be prepared for major delays

43

Lessons Learned

• Need for forms in Spanish

• Be prepared for varied knowledge levels– Including data issues

• Promote data utilization

44

Lessons Learned:Agency level data use

• Clearly identify the intended users of the data

• Develop buy-in among data stakeholders

• Identify evaluation questions meaningful to the intended users

45

Lessons Learned: TA and Training Issues

• Regional trainings require out of county travel

• Training design must support new and experienced users of ELI

• SOW vs ELI vs Real Life

• Site visits often perceived as punitive

46

Next Steps

• Outcome evaluation

• Continue on site agency level TA on intervention set up

• Data Use– More training required– Report manual

47

Next Steps

• Suggestions for next steps– What should be the focus of future ELI

trainings?– How can we make ELI more agency friendly?

48

Contact info

• Shanna Livermore, Office of [email protected]

916-449-5823

• Jeffrey Bernstein, CAPS/[email protected]

1-800-983-5550