Doug Piper Brooke Blaalid Manda Walters Population Health Institute School of Medicine and Public Health University of Wisconsin – Madison Webinar 10 January.

Doug PiperBrooke BlaalidManda Walters

Population Health InstituteSchool of Medicine and Public Health

University of Wisconsin – Madison

Webinar 10 January 2012

Population Health Institute - University of Wisconsin - Madison

SOUTH DAKOTA’S SFP SIG Project Evaluation

CSAP’s Evaluation Requirements: Community Outcomes

10 January 2012 1


CSAP’s Evaluation Requirements: Community Outcomes

10 January 2012 2

Today’s discussion topics:

1.An overview of South Dakota SPF SIG evaluation elements

2.Focus on the Community Outcome data reporting requirements

3.Specifics of what UW PHI needs to know about each Coalition’s Community Outcome data

4.Phone conversations to record that information

5.After the main webinar topic – discussion on collecting additional PLI data.

QUESTIONS?


10 January 2012

3

Indicators reflect a conceptual operationalization of the targeted priority, but an indicator is not the actual measure. In this project may be that there are multiple indicators for a given priority. For example, if a coalition is targeting the prevention of underage drinking as a priority, both 30-day use and age of first use might indicators of this priority. There are 3 types of indicators:Consumption patterns are the way in which people drink.

Consequences are the social, economic and health problems associated with the use of alcohol use and abuse, such as addiction, criminal activity, drinking and driving, etc.

Intervening variables are factors that have been identified as being strongly related to - and influential in - the occurrence and magnitude of substance use problems and consequences. Intervening variables are individual traits or experiences which drive their behaviors, such as risk factors and protective factors.

Environmental prevention strategies attempt to change the surroundings in which people choose to use, or not use, substances. They aim at changing environmental conditions that contribute to substance abuse. These strategies often involve changing laws, policies, and regulations to make it more difficult for people to abuse substances.

Individually-focused prevention programs usually endeavor to change individuals within the community or society by influencing risk and protective factors.

Glossary


10 January 2012

4

Glossary (page 2)

Measure pertains to the survey item used to operationalize the indicator.

An example for coalitions using SURVEY DATA and having a priority of prevention underaged drinking, an indicator might be 30-day use of alcohol. The measure is “During the past 30 days, on how many days did you drink one or more drinks of an alcoholic beverage?”

An example for coalitions using SURVEILLANCE DATA and having a priority of drinking and driving, the measure might be alcohol-related auto crashes occurring between 10 p.m. and 6 a.m.

NOMs – the Congressionally required National Outcome Measures. Reporting on one of the NOMS is required of all coalition receiving SPF SIG funds.

Surveillance system data are statistics, records or archives used to track trends of characteristics in your community’s population. Examples of surveillance systems are enumerating events of interest to units of government – such as county health departments, schools or sheriff’s offices. In our case they are most likely to be used as indicators of environmental efforts and changes.

Survey – information taken from local or state questionnaires, whether by telephone, paper-and-pencil or web sites, which represent a specific population’s characteristics, behaviors or views.

5

The 5 SPF Steps and Coalition Work Flow

COALITION’S COMPREHENSIVE STRATEGIC PLAN

Implement EBPs Build coalition-area capacity

Increase coalition-area support of SPF

Step 4: Community capacity and mobilization

10 January 2012

a. Determine evaluation components (design)

b. EST. A DATA COLLECTION PLAN CLI Pts 1 & 2 PLI for individual EBPs COMMUNITY OUTCOMES Unique components

c. Analyze and interpret data

d. Use the evaluation results

Mobilize community/ coalition area’s support for SPF SIG priorities


Access data on substance use and related consequences

Identify underlying causes Decide on the Coalition’s priorities

Do a gap analysis of prevention services

Conduct an environmental scan of the local substance abuse-related system

Choose which EBPs (interventions) to

implement

Increase community capacity to achieve SPF SIG priorities

Flow Chart of Required Data Elements

10 January 2012


6

Type of EBP

(inter-vention) chosen

Type of EBP

(inter-vention) chosen

OUTCOME OBJECTIVEMEASURES1. Participant-level data (PLI)2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.

OUTCOME OBJECTIVEMEASURES1. Participant-level data (PLI)2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.

OUTCOME OBJECTIVE MEASURES1. EBP-related outcome data: changes in policy, compliance checks, social access.2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.

OUTCOME OBJECTIVE MEASURES1. EBP-related outcome data: changes in policy, compliance checks, social access.2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.

Coalition-related objectivesCoalition-related objectives Program-related objectivesProgram-related objectives

Individual-focused

intervention

Individual-focused

intervention

PROCESS OBJECTIVE MEASURES1. CLI pt. 2 – intervention strategies2. South Dakota’s KIT reporting3. Archives – material developed

PROCESS OBJECTIVE MEASURES1. CLI pt. 2 – intervention strategies2. South Dakota’s KIT reporting3. Archives – material developed

PROCESS OBJECTIVE MEASURES1.CLI pt. 1 – Coalition Partner Activities2.UW PHI records and archives

PROCESS OBJECTIVE MEASURES1.CLI pt. 1 – Coalition Partner Activities2.UW PHI records and archives

OUTCOME OBJECTIVE MEASURES1.Coalition Capacity Checklist (completed)2.Successful completion of EBPs with target pop.

OUTCOME OBJECTIVE MEASURES1.Coalition Capacity Checklist (completed)2.Successful completion of EBPs with target pop.

Environ-mental

intervention

Environ-mental

intervention

SPF SIG CoalitionSPF SIG Coalition

QUESTIONS?

Community Outcome Data (p. 1)

10 January 2012


7

Community epidemiological outcome data Data collected representative of the population in the Coalition’s service

area Generally gathered through local sources because existing national

sources may not include the correct measure or sample the correct geographic area

Related to SPF SIG targeted priorities and NOMs

Community outcome data should: Relate to the community’s targeted priorities Include at least one NOM for your service area Reflect the areas where SPF SIG programs are implemented

Community outcome data can: Include multiple priorities, indicators, and measures Include intervening variables

What is purpose: to assess the impact of EBPs (interventions) on priorities for the entire service area: the data is parallel to the coalition’s service area.

How is the data collected: As appropriate to the secondary data source being reported.

Who collects:

• The state evaluator (UW PHI) and the State Epidemiological Outcomes Workgroup (SEOW) will make as much data available to the local evaluators as is pertinent.

• Much of the Community Outcome data will be collected from local sources and forward to UW PHI (e.g. local school district data, sheriff’s department data).

• Local evaluators and UW PHI will cooperate to collect pertinent information.

• The UW PHI then puts the Community Outcome data into the MRT.

How often is data reported: The baseline will be from a period prior to implementing the EBP (intervention) and secondary data will be updated as it becomes available. Primary data will be reported as it becomes available.

10 January 2012


8

Community Outcome Data (p. 2)

Review of the NOMs

10 January 2012Population Health Institute - University of

Wisconsin - Madison9

Domain: Reduced Morbidity - Abstinence from Drug Use/Alcohol Use

30 Day Use Perceived Risk/Harm of Use

Age at First Use Disapproval of Substance Use (youth only)

Domain: Employment/ Education

Perception of Workplace Policy

Daily School Attendance

Domain: Crime and Criminal Justice

Driving While Under Influence of Alcohol

Alcohol-Related Traffic Fatalities

Alcohol and Drug-Related Arrests

Domain: Social Support/ Social Connectedness

Family Communication Around Drug Use

QUESTIONS?

10 January 2012


10

Coalitions gather local community

epi data IF necessary

and/or possible

UW PHI gathers data: From national sources (if available) From SEOW From Coalitions

DSS Prevention Program reviews

data entered by UW PHI and submits to

CSAP via MRT

Preliminary data entered into MRT for SD Prevention Program’s review

SD SEOW sends pertinent

epidemiological data to UW PHI

Finalized data is accepted by DITIC for CSAP

Community Outcome Data Flow

Coalition’s own data is available via

MRT download

QUESTIONS?

Overview of Community Outcome Data Necessary for CSAPPRIORITY AND TARGET POPULATION

Details of surveillance or event data for the MRT:1) Data source; who collected & when2) Population on which event data is based3) Geographic unit of the event4) Census of population 5) N of the population on which the event is based6)Event census population characteristics.7)How is the outcome measure calculated?

Details of survey data for the MRT:1) Data source; who collected & when2)Exact wording of the question3)Response options4)Sampling strategy5)Definition of the sample6)N of the population7)Demographic characteristics of the respondents8)Specifics of value reported as an outcome

Start of EBP – did it start prior to the SPF SIG funding or prior to the data collection?

Surveillance or event dataExisting data orNewly collected data

Survey dataExisting data orNewly collected data

An indicator reflects a conceptualization of the priority.

Type of data - consumption patterns; consequences of use; or intervening variables

Is it a NOMs measure? Is it a NOMs measure?

Individually-focused EBPsEnvironmental EBPs EBP implemented

QUESTIONS?

Discussion Guide for Baseline Data – Both Forms

10 January 2012


12

BASIC INFORMATION

1. Coalition name and location

2. Priority

3. Indicators reflect a conceptual operationalization of the targeted priority, but an indicator is not the actual measure.

4. Is there more than one Community Outcome indicator for this priority? IF YES, what are the others?

[The PHI will need this information for each indicator entered into the federal reporting system.]

5. Indicator type: Consumption patterns. Consequences Intervening variables

6. Are you reporting survey data? IF YES, PLEASE SWITCH TO THE SURVEY FORM.

Are you reporting surveillance system data? IF YES, PLEASE CONTINUE BELOW.

Discussion Guide for Baseline Data - Surveillance

10 January 2012


13

DEFINING THE SURVEILLANCE DATA

7. Data source:

Which surveillance system are you using as an indicator? Who or what agency collected the data? When were these data most recently collected? How often is this data collected? Who or what agency stores the data? Is the data accessible on-line?

8. What SPF SIG-funded EBP(s) or intervention addresses this priority?

9. EBPs implemented prior to data collection?

Were any SPF-funded interventions implemented in this community prior to this data collection?

10. Starting date for EBPs related to this priority

During what quarter do you expect EBPs related to this priority to be first implemented in this community?

11. A measure is the data item used to operationalize the indicator.

As an example, for coalitions with a priority of drinking and driving – and using SURVEILLANCE DATA as outcomes – the measure might be alcohol-related auto crashes occurring between 10 p.m. and 6 a.m. in the target county.

12. NOMs itemDoes this measure qualify as a NOMs item? Do you have, or expect, a waiver from CSAP regarding the

NOMs to be reported?

13. Connection between priority and measure

What is the logical connection between your priority and objectives and this indicator and measure?

Discussion Guide for Baseline Data – Both Forms

10 January 2012


14

CHARACTERISTICS OF THE POPULATION IN THE SERVICE AREAON WHICH THE SURVEILLANCE DATA IS BASED [This usually encompasses a larger population than is

included in the surveillance data]

14. Population on which surveillance data is based

Who constitutes the population for the surveillance data being reported for this measure?

15. Geographic unit of surveillance or event

What is the geographic unit of the data being reported?

16. Surveillance census population age range: maximum and minimum

What are the maximum and minimum ages of the population on which the surveillance data is based?

17. N of population for surveillance data

How many persons comprise the population on which the surveillance data is based for this measure?

Demographic data of the population from which surveillance data comes (if known)

18. Gender % Female % Male

19 Hispanic Ethnicity % Hispanic/ Latino % Non-Hispanic/Latino

20. Race % Asian% Native Hawaiian or other Pacific Islander

% Multiracial

% American Indian or Alaska Native

% Black or African American

% White % Other

Discussion Guide for Baseline Data - Surveillance

10 January 2012


15

OUTCOME DATA

21. Reported outcome for surveillance or event data

What is the result you are reporting for this measure?

22. Number of eventsHow many times did this event occur in the community?

23. What format will the data be in when reporting to the UW PHI?

QUESTIONS?

Discussion Guide for Baseline Data – Survey Data

10 January 2012


16

DEFINING THE SURVEY

7. Data source:

Which survey are you using to collect data for this indicator? Who or what agency collected the data? When were these data most recently collected? How often is this data collected? Who or what agency stores the data? Is the data accessible on-line?

8. What SPF SIG-funded EBP(s) or intervention addresses this priority?

9. Were any SPF SIG-funded EBP(s) implemented in this community prior to the dates during which the survey data were collected?

During what quarter were EBPs related to this priority first implemented in this community?

10. A measure is the data item used to operationalize the indicator.

Source item For which survey item are you reporting data? What is the exact question that measures your NOM? (exact wording)

Response options What are the response options for the survey item used for this measure? (exact wording)

11. NOMs Item Does this measure qualify as a NOMs item? Do you have, or expect, a waiver from CSAP regarding the

NOMs to be reported?12. What is the logical connection between your priority and objectives and this indicator and measure?

13. What format will the data be in when reporting to the UW PHI?

BASIC INFORMATION (same as for surveillance data)


10 January 2012


17

POPULATION FROM WHICH THE SURVEY DATA WAS TAKEN

14. Population from which survey sample is drawn

From what population was the survey sample drawn?

15. Population age or gradeWas the survey population defined by age or grade level?

16. Survey population age range minimum and maximum

What were the minimum and maximum ages of members of the survey population?

17. Survey population grade range What grade levels were targeted by the survey?

18. N of populationApproximately how many people were in the survey population?

Survey Sample Information

19. Sampling Strategy What sampling strategy was used for this survey?

20. N of SampleHow many persons were drawn as part of the survey sample?

21. Number of Respondents How many persons participated in the survey?

22 – 24. Demographic characteristics (if known) of the targeted population from which the survey is drawn. (SAME CATEGORIES AS PREVIOUSLY DISCUSSED)


10 January 2012


18

OUTCOME DATA

25. Describe the survey data reported as an outcome.

Provide a brief description of the outcome being reported with this measure and, thus, being used to track changes in the related indicator. Please be specific about the behavior being assessed.

26. Value typeType of number reported (e.g., percentage, mean, rate) in the estimated value.

27. Statistical details Standard error Standard deviation

95% confidence interval

28. Number of valid responses for this specific survey item

How many respondents provided a valid response (i.e., not missing) to the survey item of interest?

29. Possible #of valid responses for specific survey item

How many respondents could have provided a valid response to the survey item of interest?

30. Denominator definitionHow is this measure’s denominator defined for calculation?

31. Denominator valueWhat is the value of the denominator for this measure calculation?

32. Calculated value of the outcome measure.

The reported outcome pertains to the actual result and related statistic you are reporting for this measure. What is the value of this measure for the community?

33. Validity comments or concerns

Please comment on any concerns about how well each measure provides an accurate and unbiased assessment.

34 – 36. Demographics of the survey respondents (SAME CATEGORIES)QUESTIONS?

Some Coalitions will need to send local data to UW PHI depending upon their EBPs (interventions) and their evaluation design. This will be worked out with each Coalition during technical assistance sessions.

Conversations with the national evaluators of the SPF SIG indicate that the most problematic areas for other states and coalitions have been:

Finding or collecting community outcome data corresponding to the Coalition’s service area.

Finding or collecting data corresponding to the target population – especially the 18 to 25 age group.

Ensuring that there is a logical or theoretical link between the community outcome data collected and the EBP (intervention) being implemented.

Ensuring that data is collected and reported consistently from year to year and across all targeted sub-populations.

10 January 2012


19

Community Outcome Data:

Potential Pitfalls

QUESTIONS?

Help Using the Reporting Community Outcomes Data via the MRT

10 January 2012


20

Technical difficulties or questions on the use of the web-based data entry system:DITIC Help Desk

1-888-DITIC-4-U (1-888-348-4248)[email protected]

Questions specific to the content for South Dakota:

Doug PiperSD SPF SIG Evaluation Director(608) [email protected]

Sarah Linnan SD SPF SIG Evaluation Data-base Manager(608) [email protected]

Brooke Blaalid, MSWSPF-SIG Evaluator/Trainer for Eastern SDHome office (605) 689-0478 Cell (605) [email protected]

Manda Walters, MSSPF-SIG Evaluator/Trainer for Western SDHome office (605) 791-0259Cell (605) [email protected]

Doug Piper Brooke Blaalid Manda Walters Population Health Institute School of Medicine and Public Health University of Wisconsin – Madison Webinar 10 January.

Documents

use of alcohol use

health problems

wisconsin madison10

priority of drinking

day use of alcohol

community outcomes10

targeted priority

given priority