Evaluating the Impact of the Advancement Via Individual Determination Program on Ninth-Grade Students' Learning and Study Skills Jenny Nagaoka, Melissa.

Evaluating the Impact of the Advancement Via Individual Determination Program on Ninth-Grade

Students' Learning and Study Skills

Jenny Nagaoka, Melissa Roderick, and Melanie LaForce

June 29,2010

2010 IES Conference

What is AVID?

• History:– Began in 1980 in San Diego, CA– National AVID office was created to provide standards and oversight

in implementation and expansion– 2009-2010 school year: nearly 4,500 schools in 45 states

• AVID in the Chicago Public Schools (CPS):– Part of a larger initiative to improve student preparation for and

access to postsecondary education

– First implemented in CPS in 2003-2004 in 8 high schools

– Expanded to 42 high schools by 2007-2008 and 59 high schools currently, including 2 all-AVID high schools.

The AVID Program

Program & Goals:• School-based program, provides academic and social

support• Designed to improve the preparation and college access of

middle-achieving students

The AVID Elective Class- “How to be a student”Curriculum around Supports & Skill Building:• Academic – organizational, note-taking, writing, critical

thinking skills, and questioning skills• Social – peer support, teacher guidance, college advising

The AVID Site Team• Site Coordinator/Teacher; AVID Administrator; English, math,

science, social studies teachers

Research on AVID

• Some positive evidence– More likely to be enrolled in college (Mehan, et al., 1996 )– Schools with AVID have higher graduation rates, advanced

course-taking (Watt, et al., 2006)– But do not address selection effects

• SRDC British Columbia Study – Randomized Experiment– Non-urban, non-minority district

Data and Sample

• Data– Consortium on Chicago School Research data archive

• CPS school records, including high school transcripts• Biennial survey of CPS students

• Sample– First-time 9th graders in 14 CPS AVID high schools, not in special

education, enrolled in both fall and spring semesters • 7,357 students in pre-AVID cohorts (2002 and 2003)• 14,031 students in post-AVID cohorts (2005 to 2008), 2,521

AVID students and 11,510 non-AVID students • 51% female and 49% male, 68 % African-American, 23%

Latino, 10% White

1. Are AVID students doing better than other students on critical 9th grade outcomes that predict high school and college graduation?

2. Are these differences due to selection effects?

Research Questions

Does AVID have an impact on 9th grade outcomes?

• Supports & Skill Building– Academic: study habits– Social supports: peer support, sense of belonging, teacher-

student trust, teacher support, engagement

• Academic outcomes– Cumulative weighted GPA and probability of 3.0 or higher

GPA– On-track– Absences– English GPA and probability of B or better (English I)– Math GPA and probability of B or better (Algebra or geometry)

Dealing with Selection Bias

1. Propensity score matching– Estimate probability for treatment based on measured

variables• Drawbacks: Only matching on measured characteristics (test

scores, demographic characteristics, fixed effects for elementary school, 7th and 8th survey responses)

– Sample of students in 14 AVID schools is divided into deciles based on propensity scores

Dealing with Selection Bias

2. Simulated Propensity Scores– Pre-post comparison of treatment constructed by simulating a

comparison group of students who have a high propensity for AVID in the pre-AVID period

– Apply propensity model coefficient weights and simulate probability for AVID in students before AVID was implemented

– Three groups: pre, post-period AVID, post-period not in AVID• Pre- and post non-AVID should be similar once we control for

background characteristics and cohort effects.

– Sample is divided into deciles based on propensity scores with same cut points for pre and post cohorts.

Average GPA for 9th grade AVID and Non-AVID students

Post-AVID cohorts 2003-2008

Probability of having a B or better GPAfor 9th grade AVID and Non-AVID students

Post-AVID cohorts 2003-2008

On-track and absences for 9th grade AVID and Non-AVID students

Post-AVID cohorts 2003-2008

9th grade on-track rates Days absent per semester in 9th grade

Allensworth & Easton (2007) What matters for staying on-track and graduation from high school in Chicago Public Schools

Propensity Score Matching 9th grade academic outcomes for Post-AVID cohorts 2003-2008

Decile 7 Decile 8 Decile 9 Decile 10

Core GPA 0.11 0.15* 0.17** 0.08

Probability of 3.0+ GPA

0.15 0.36* 0.40* 0.05

English GPA 0.10 0.27** 0.22* 0.26**

Probability of B in English

0.18 0.23 0.38** 0.31*

Math GPA -0.05 0.21 0.27† -0.09

Probability of B in math

0.15 0.09 0.16† 0.01

Log Absences -0.12 -0.23** -0.19* -0.19**

On-track 0.32* 0.14 -0.01 -0.03

Simulated Propensity Scores9th grade academic outcomes for Pre-AVID cohorts (2001 and 2002)

and Post-AVID cohorts (2003-2008)

Decile 7 Decile 8 Decile 9 Decile 10

Core GPA 0.15* 0.16* 0.10 -0.00

Probability of 3.0+ GPA

0.29 0.35† 0.36* -0.21

English GPA 0.07 0.22* 0.14 0.11

Probability of B in English

0.18 0.14 0.22 0.23

Math GPA 0.26** 0.13 0.07 -0.02

Probability of B in math

0.20 0.29 0.13 0.07

Log Absences -0.12 -0.22*** -0.23** -0.13*On-track 0.23 0.00 -0.22 -0.141

9th grade study habits for Post-AVID cohorts 2003-2008

How much do you agree with the following:

• I set time aside to do my homework and study.

• I try to do well on my schoolwork even when it isn’t interesting to me.

• If I need to study I don’t go out with my friends.

• I always study for tests.

Propensity Score Matching 9th grade academic skill and social outcomes for Post-

AVID cohorts 2003-2008

Conclusions

• AVID is clearly doing a good job of selecting students

• AVID students are performing poorly in school – Average GPA is <2.0

• Lack of Strong Effects may be linked to differences between CPS and other AVID districts– Implementation issues– “Middle Achieving” may mean different things– Teaching study skills may not translate to large urban districts

• AVID focuses on improving student skills without changing what is happening in the classroom.