USING SECONDARY DATA TO HELP ASSESS -

John J. Green The University of Mississippi

Center for Population Studies

State Data Center of

Mississippi - -

Annual Affil iates Meeting

-- November

2012

USING SECONDARY DATA TO HELP ASSESS AND EVALUATE

COMMUNITY, ECONOMIC, AND HEALTH DEVELOPMENT

PROGRAMS

  Community, economic, and health development programs are often planned and implemented without systematic prior assessment

  Additionally, monitoring and evaluation of these programs is often lacking

  Insufficient data and limited resources pose problems   This results in a knowledge gap concerning: what problems are in

need of attention, the factors contributing to them, what strategies work/do not work in addressing the problems, and the improvements that need to be made

  Secondary data from public sources may help to fill some of these gaps

  Mixed-methods designs that combine secondary data with primary data offer a practical strategy for conducting efficient and useful assessments and evaluation

  In all cases, creative, careful, and logical design is of critical importance

OVERVIEW OF THE ISSUES

  Case Design: Focuses on one case (group, organization, community, event) within its social context; can also do multiple case comparisons

 Cross-Sectional Design: Data are collected for all of the variables of interest using one sample at one time

  Longitudinal Design: Data are collected at two or more times, including trend, cohort, and panel studies

  Experimental Design: The independent variables are controlled in some way and subjects/participants are randomly assigned to experimental and control groups

 Quasi-Experimental Design: Models experimental design, but is missing one or more aspects of a “true” experimental study, most frequently random assignment into experimental and control groups

ASSESSMENT AND EVALUATION DESIGN

Adapted from: Adler, Emily S. and Roger Clark. 2007. How it’s Done: An Invitation to Social Research. Belmont, CA: Thomson/Wadsworth.

  In terms of arguing causality (that program X leads to outcome Y), experimental design is often considered the “gold standard”

  However, it is typically not possible to meet the requirements of

traditional true experimental design for community, economic, and health development programs

  Furthermore, there are many nuances and complexities to social life that may be overlooked with traditional experimental design

  So, the challenge is to design assessment and evaluation studies that:   Provide a logical basis to address influence/causality   Can be conducted within real world constraints   Take into account the complexities of social life

  A combination of logical reasoning based on quasi-experimental design and use of mixed-methods might help

THE DESIGN CHALLENGE

  “Primary” data are collected first-hand by the researcher for the specific purpose of a particular project  Observation

  Interviews

  Focus groups

  Surveys

  “Secondary” data are collected for a different and/or broader purpose (typically from administrative records, vital records, and surveys), but are available for other uses We sometimes use “available” data as a synonym for “secondary” data

  This presentation focuses on secondary data use, but in the context of program assessments and evaluations with mixed-method designs

PRIMARY AND SECONDARY DATA

 Decennial Census, Economic Census, Census of Agriculture

 American Community Survey

 Vital health records

  Educational data

 Organizational/programmatic administrative records

 Most secondary sources involve periodic data collection, allowing for longitudinal analysis

  Both a benefit and challenge to secondary data sources like these are their ties to specific geographic spaces (states, counties, etc.)

COMMONLY USED SOURCES OF SECONDARY DATA

  AmeriCorps*VISTA   The AmeriCorps*Volunteers In Service to America

(VISTA) program is a national initiative to “fight poverty with passion”

  Funded by the Corporation for National and Community Service

  Volunteers partner with nonprofit agencies and educational organizations to help build capacity to better serve communities and, overtime, address the factors influencing poverty

  Delta State University and the University of Mississippi both operate VISTA programs, and it is imperative that leaders, community partners, and volunteer members understand the poverty levels and trends in the places where they are working

  Additionally, monitoring and evaluation efforts are necessary to track their understanding of poverty and how the work they are doing builds the capacity of local organizations to provide needed services

EXAMPLE: EDUCATION, POVERTY, AND WORKFORCE DEVELOPMENT PROGRAMS

  Tri-County Workforce Alliance   A nonprofit focused on expansion and enhancement of educational and workforce

development opportunities in Bolivar, Coahoma, and Quitman Counties of Mississippi, the Tri-County Workforce Alliance (TCWA) is also serving residents of Sunflower and Tallahatchie Counties

  Among its programs, TCWA is partnering with the Dreyfus Health Foundation and Mississippi Office of Nursing Workforce on the “New Pathways” program (with funding from the WK Kellogg Foundation) to offer educational opportunities to youth for health care careers   Youth preparation and mentorship   Health careers academy   Professional and leadership development workshops   Student nurse externships and dedicated educational units   Problem Solving for Better Health (PSBH) workshops   Education and advocacy in the core component of children’s health and development in the

context of disparities

  This work requires assessment to understand education, employment, and poverty trends; and there is need for monitoring and evaluation

EDUCATION, POVERTY, AND WORKFORCE DEVELOPMENT PROGRAMS (CONT’D)

  Primary Data Sources:

  In-depth interviews (VISTA)

  Participatory group processes (VISTA and TCWA)

  Fish bowl

  World café

  Surveys (VISTA and TCWA)

  Secondary Data Sources:

 Decennial Census

 American Community Survey

 Mississippi Department of Education

EDUCATION, POVERTY, AND WORKFORCE DEVELOPMENT PROGRAMS (CONT’D)

0

20

40

60

80

100

% a

t or B

elow

Pov

erty

Lev

el

Family Poverty in Mississippi and the Delta Region (2006-2010)

0

20

40

60

80

100

% B

ache

lor’s

Deg

ree

or H

ighe

r Educational Attainment (% of Adults 25+ with a Bachelor’s Degree or Higher) in

Mississippi and the Delta Region (2006-2010)

Source: U.S. Census Bureau, 2006-2010 American Community Survey. Figures constructed by The University of Mississippi Center for Population Studies. Margins of error are based on 90% confidence intervals.

Mississippi Bolivar Coahoma Quitman Sunflower Tallahatchie

Total income below poverty

16.1 (+/- 0.2)

26.4 (+/- 2.0)

26.9 (+/- 2.8)

29.2 (+/- 4.2)

27.5 (+/-2.8)

26.6 (+/- 4.1)

Male Female Male Female Male Female Male Female Male Female Male Female

Total income below poverty by sex

36.8 63.2 33.7 66.3 35.2 64.8 39.0 61.0 35.3 64.7 43.9 56.1

Mississippi Bolivar Coahoma Quitman Sunflower Tallahatchie

Male Female Male Female Male Female Male Female Male Female Male Female

< High school degree 44.1 37.3 50.9 43.9 49.3 39.5 62.1 46.1 48.6 38.3 31.8 45.8

High school or GED 32.3 33.0 30.4 29.7 33.2 26.5 17.3 30.8 29.1 28.2 48.2 26.8

Some college or associate’s

17.8 24.4 10.0 21.3 16.6 28.3 19.8 21.4 19.0 25.5 18.8 21.8

Bachelor’s or above 5.8 5.3 8.7 5.1 0.9 5.7 0.8 1.7 3.3 8.0 1.2 5.6

Percent of Adults 25+ with Incomes Below the Poverty Line in Mississippi and Select Delta Region Counties (2006-2010)

Percent of Adults 25+ with Incomes Below the Poverty Line by Sex and Educational Attainment in Mississippi and Select Delta Region Counties (2006-2010)

Source: U.S. Census Bureau, 2006-2010 American Community Survey. Tables constructed by The University of Mississippi Center for Population Studies. Margins of error are based on 90% confidence intervals. Margins of error not shown here are available from the U.S. Census Bureau’s American Factfinder website.

18.4 17.9 15.6 16.9

14.9 14.9 15.7 16.0 15.5 14.9 14.9 15.7 15.6 16.1 15.3 18.7

0 4 8

12 16 20 24 28 32 36

Aver

age

Com

posi

te A

CT

Scor

es

Average Composite ACT Scores in Mississippi, Selected Delta Region School Districts, and Program Participants (2011)

Sources: Mississippi Department of Education, Statewide Assessment Information (2011) and Tri-County Workforce Alliance program documentation (2011). *Students involved in high school nurse mentorship program; participants are from Bolivar, Coahoma, Quitman, and Tallahatchie counties.

  The Right! from the Start program is focused on conducting research, building awareness, and providing education and training on the topic of poor birth outcomes, especially preterm births

  Coordinated by Women and Children’s Health Initiatives and the Community Foundation of Northwest Mississippi, with support from the WK Kellogg Foundation

  Data sources:   Primary – Participatory community forums (world café

sessions)   Upcoming will include survey of workshop participants

  Secondary –   Decennial Census

  American Community Survey

  Mississippi Department of Health Vital Records and Statistics

EXAMPLE: MATERNAL AND CHILD HEALTH PROGRAM

0

5

10

15

20

25

30

1980 1985 1990 1995 2000 2005 2010

Pret

erm

(<37

Wee

ks)

Birth

Rat

e Pe

r 100

Liv

e Bi

rths

Preterm Birth Rates in Mississippi, 1980 to 2011

0 5 10 15 20 25 30 35

10 - 14 years

15 - 17 years

18 - 19 years

20 - 24 years

25 - 29 years

30 - 34 years

35 - 39 years

40 years +

Preterm Births (<37 Weeks) Per 100 Live Births

Preterm Births by Race and Age, 2007-2011 Pooled Data

White

Black

Total

Source: Mississippi Department of Health, Vital Records. Figures constructed by the University of Mississippi Center for Population Studies.

  There are challenges confronted in using secondary data as part of evaluation design

 Data limitations abound, as the secondary sources were not developed with the purposes of evaluating specific community-based initiatives

  Programs often serve a limited number of people (especially at their early stages) and population level changes are unlikely to be observed, even for the most successful efforts

  Programs often take place in towns, counties, and regions with persistent education and poverty problems, so even if a program is successful, it may be doing so in a worsening context

  It takes a long-term view to see marked population level changes around education, workforce development, poverty, and health problems, yet funders want to see shorter-term outcomes

CHALLENGES

  Program assessment and evaluation are of critical importance to effective decision making, especially in the face of limited resources

  Educational institutions, government agencies, and private foundations can work with nonprofit organizations to better assess the needs for programs, monitor progress, and evaluate processes and outcomes

  Methodological innovations are needed in regard to design, data collection, and analysis   Expand use of quasi-experimental design

  Longitudinal focus (investigate patterns over time)

  Use of comparison groups (compare people, groups, and/or places with a program to those that were similar, with the exception of having the program)

  Mixed methods, including quantitative and qualitative approaches   Statistical data are helpful for identifying and describing patterns

  Observation, interview, and focus group data are useful for interpreting patterns and informing propositions to explain patterns

MOVING FORWARD

 Faculty, staff, and students affiliated with The University of Mississippi Center for Population Studies and the Department of Sociology and Anthropology

 Aaron E. Henry Community Health Services Center  Community Foundation of Northwest Mississippi  Dreyfus Health Foundation  Mississippi Office of Nursing Workforce  Tri-County Workforce Alliance  Women and Children’s Health Initiatives  WK Kellogg Foundation

SPECIAL THANKS