Sean P. Corcoran Lori Nathanson James Kemple New York University INVALSI – Rome, Italy – October 2012 T HE I MPACT OF H IGH S CHOOL C HOICE ON M EDIATORS.

Sean P. CorcoranLori NathansonJames Kemple

New York University

INVALSI – Rome, Italy – October 2012

THE IMPACT OF HIGH SCHOOL CHOICE

ON MEDIATORS OF STUDENT SUCCESS

HIGH SCHOOL CHOICE IN NYC

80,000 rising 8th graders participate in high school choice each year

700 high school programs available

12 choices, students rank in order of preference

1 matched school for 86% of high school admissions process (HSAPS) participants(main round in 2008 and 2009)

Preliminary, not for quotation or distribution.2

HIGH SCHOOL CHOICE IN NYC

Source: http://www.nytimes.com/schoolbook/2011/09/30/high-school-admissions-choice-but-no-equity/


RATIONALES FOR CHOICE

To provide opportunities for students in low-performing, racially and/or economically isolated schools

To facilitate matches between students and schools

To engage students and their families in the process of choosing a school

To promote competition, encourage innovation, and incentivize high performance


LITERATURE ON SCHOOL CHOICE

Preliminary, not for quotation or distribution.

A large literature has examined the effects of choice in the context of open enrollment, charter schools, and voucher programs (e.g., Cullen, Jacob, & Levitt, 2006; Abdulkadiroglu et al., 2009; Zimmer et al., 2011; Hastings & Weinstein, 2008)

These are typically voluntary (“opt-out”) or targeted programs

5

LITERATURE ON SCHOOL CHOICE


Most of these papers are interested in effects on achievement, and the focus is often on school quality differences (e.g., Deming et al., 2009; Lai, 2007; Lauen, 2009; Pop-Eleches & Urquiola, 2011; Clark, 2010)

There is a growing literature that examines effects on “non-cognitive” outcomes, and achievement not captured by test scores (Imberman, 2011; Booker et al., 2011; Pop-Eleches & Urquiola, 2011; Deming et al., 2009; Hastings et al. 2012)

6

THIS PAPER


We examine whether admission to a desired school impacts mediators of student achievement – indicators of student engagement in four broad categories

7

STUDENT ENGAGEMENT WITH SCHOOL

Multiple dimensions (e.g., Fredricks et al., 2004) Behavioral: attendance, participation in extracurricular

activities

Cognitive: perceptions about value of school, investment in learning

Psychological/Emotional: connections with teachers, peer culture

Academic: completing required tasks (courses, exams)


STUDENT ENGAGEMENT WITH SCHOOL

Why might choice have implications for engagement? Differences in school quality, practices, proximity Better match between students and schools (Stage-

Environment Fit Theory, Eccles et al., 1993) Promotes autonomy and investment (e.g. Connell’s

self-system model, 1990)

There are potentially both school and match/fit effects of choice on engagement


THIS PAPER


We provide an initial look at differences between the engagement of students who are successfully matched to their first choice school and students who are not

Additionally, we look at differences within schools in the engagement of students matched as 1st choice vs. other lower choices

10

THIS PAPER


We find significant effects of receiving a first choice on many measures of engagement, using a subsample of students who were subject to random assignment

Within schools, however, the differences are much smaller, suggesting differences in engagement are attributed to different school assignments rather than match/fit

11

DATA

HSAPS data (about 80,000 students per year) matched to NYCDOE administrative data

2008 and 2009 cohorts of 8th graders (N=133,387) Matched through HSAPS Did not opt out or enroll in a specialized exam HS (e.g.

Stuyvesant) Enrolled in a public high school in fall of 9th grade


THREE SAMPLES


1. All students meeting the above criteria2. All students whose first choice was an

oversubscribed, nonselective program (e.g., Bloom et al., 2010)

3. All students who were matched to an oversubscribed, nonselective program, but not necessarily their first choice

13

HOW STUDENTS ARE ASSIGNED TO SCHOOLS


HOW STUDENTS ARE ASSIGNED TO SCHOOLS


PROGRAMS BY ADMISSIONS METHOD

Admissions Method # (%) of Programs

2008 2009

Screened 193 (29%) 197 (29%)

Educational option 197 (29%) 192 (28%)

Limited unscreened 167 (25%) 179 (27%)

Audition 68 (10%) 63 ( 9%)

Zoned 32 ( 5%) 32 ( 5%)

Unscreened 16 ( 2%) 15 ( 2%)

All HS Programs 673

(100%)

678

(100%)

Note: each program has different admissions criteria that determine the matching process. Students can rank any combination of programs for their 12 choices.


LIMITED UNSCREENED PROGRAMS

Priority to students demonstrating interest in the school: attending a school’s Information Session or Open

House visiting the school’s exhibit at a High School Fair

When oversubscribed, HSAPS awards seats at random


STUDENTS’ CHOICES AND MATCHES

Students made an average of 7 choices. 52% of students received their first choice. 77% of students received one of their top three choices.


0.2

.4.6

.81

1 2 3 4 5 6 7 8 9 10 11 12

1st 2nd 3rd 4th 5th 6-8th

9-12th Supp 1st Supp 2nd Supp 3-12th Unmatch

18

OUTCOME MEASURES


Behavioral engagement:

Attendance rate (9th grade)Extracurricular activity participation

Cognitive engagement:

Single scale based on 12 survey items

Psychological engagement –relationships with teachers:


Psychological engagement – perceptions of peer culture:


Academic engagement:

Credits attempted and earnedRegents exams attempted and passed

20

BEHAVIORAL ENGAGEMENT


Mean

SD

Percent of days in attendance, 8th grade 90.8 9.8Percent of days in attendance, 9th grade 89.1 15.7Percent of days in attendance, 10th grade (2008 only)

86.2 19.3

Change in attendance 8th-9th grade -1.65 10.9Change in attendance 8th-10th grade (2008 only)

-4.50 14.5

Number of extracurricular activities 1.48 1.63Participated in any extracurricular activity 0.65 0.48Participated in two or more extracurricular activities

0.40 0.49

21

COGNITIVE ENGAGEMENT

Cognitive Engagement “I need to work hard to

get good grades at my school.”

“My teachers connect what I am learning to life outside the classroom.”

Cognitive Engagement Measure

Number of items

12

Alpha .88

Mean 3.06

SD .51Note: Based on 4-pt Likert scale


22

PSYCHOLOGICAL ENGAGEMENT

Relationships with Teachers “How comfortable are

you talking to teachers and other adults at your school about a problem you are having in class?”

Peer Culture “Most students in my

school help and care about each other.”

Relationships with Teachers Measure

Number of items

4

Alpha .79

Mean 2.64

SD .77

Peer Culture Measure

Number of items

4

Alpha .78

Mean 2.46

SD .67

Note: Based on 4-pt rating scale, 1=uncomfortable to 4=comfortable, or 1=unavailable to 4=available.

Note: Based on 4-pt Likert scalePreliminary, not for quotation or distribution.


23

ACADEMIC ENGAGEMENT

Students attempted an average of 15 credits and earned approximately 12 credits in 9th grade.

The average student passed one Regents exam by the end of 9th grade.


Mean

SD

Total credits attempted 14.5 4.4Total credits earned 11.6 5.4Total Regents exams attempted 1.5 1.2Total Regents exams passed (65+) 1.0 1.0

24

EMPIRICAL MODEL


We are interested in the causal impact of assignment to one’s nth choice high school (e.g. 1st), relative to that if one was assigned to a lower choice

(1) Yi = β0 + γZ1i + βXi + ui

E.g. Yi = engagement, Z1i = assignment to first choice, Xi = student covariates Problem: consider how Z1i might be assigned

25

EMPIRICAL MODEL: SELECTION PROBLEMS


EMPIRICAL MODEL


One approach: contrast students with same first choice

(2) Yi = β0 + α1i + γZ1i + βXi + ui

α1i = first choice effects Problem: generally, the likelihood of attending one’s

first choice can still be correlated with school quality, unobserved characteristics of students

27

WHAT WE DO IN THIS PAPER


Estimate model (2) for all students in our sample, all programs represented: the “naïve” (non-causal) model

Estimate model (2) for all students whose first choice was a non-selective oversubscribed school (and were subject to random assignment)

Estimate model (2) with school fixed effects—comparing students matched to same schools but at different preference. All students, and those matched to an oversubscribed non-selective school.

28

STUDENT CHARACTERISTICS BY SAMPLE


Sample 1:Base sample

Sample 2:Oversubscribe

dLimited

Unscreened1st choice

Sample 3:Matched to

Oversubscribed

LimitedUnscreened

Applicant from private school

4.1 4.1 4.3

Female 49.6 49.8 50.8Asian 12.1 3.7 3.7Black 34.1 41.4 44.0Hispanic 41.5 48.2 46.8White 11.6 6.0 4.7Special education 6.6 7.5 7.5English language learner 10.2 8.6 9.0Foreign born 19.0 12.6 13.3Eligible for free/reduced lunch

75.9 80.7 81.2

Attendance rate (% of days on roll)

90.8 89.9 89.8Note: Student characteristics are based on 8th grade data. All numbers represent percentages.

29

SAMPLE MEANS: 8TH GRADE STANDARDIZED TESTS


REGRESSION RESULTS – SAMPLE 1 ALL SCHOOL TYPES


Outcome 1st choice, w/o school effects

1st choice, with school effects

N

Attendance in grade 9 0.95*** 0.48*** 120,200Any activities 0.03*** 0.01 92,397Two or more activities 0.04*** 0.01 92,397Number of activities 0.13*** 0.02 92,397Cognitive engagement 0.05*** 0.02*** 93,371Psychological engagement (1)

0.04*** 0.02** 93,772

Psychological engagement (2)

0.11*** 0.004 93,888

Credits attempted 0.11*** 0.11*** 120,615Credits earned 0.22*** 0.24*** 120,615Regents attempted -0.06*** 0.01 120,615

Note: * p<0.05, ** p<0.01, *** p<0.001Each cell represents a coefficient estimate from a separate regression, where the indicated outcome variable is used. Models include student-level covariates, cohort dummies, and program type effects.

31

REGRESSION RESULTS – SAMPLE 2:OVERSUBSCRIBED LIMITED UNSCREENED SCHOOLS, 1ST CHOICE

Outcome 1st choice, w/o school effects

1st choice, with school effects

N

Attendance in grade 9 1.16*** 0.57 19,065Any activities 0.03* 0.02 14,481Two or more activities 0.02 0.01 14,481Number of activities 0.08 0.01 14,481Cognitive engagement 0.05*** -0.004 14,691Psychological engagement (1)

0.05* -0.01 14,764


0.09*** -0.01 14,791

Credits attempted 0.39*** 0.17 19,143Credits earned 0.95*** 0.44** 19,143Regents attempted 0.14*** 0.04 19,143


Note: * p<0.05, ** p<0.01, *** p<0.001 Each cell represents a coefficient estimate from a separate regression, where the indicated outcome variable is used. Models include student-level covariates and cohort dummies.

32

REGRESSION RESULTS –SAMPLE 3: MATCHED TO OVERSUBSCRIBED LIMITED UNSCREENED

SCHOOLSOutcome 4th-12th choice,

w/ school effects

4th-12th choice, w/ school

effects, cntl

N

Attendance in grade 9 -0.55* -0.80* 23,599Any activities -0.02 -0.02 17,937Two or more activities -0.02 0.001 17,937Number of activities -0.02 -0.01 17,937Cognitive engagement -0.03* 0.001 18,223Psychological engagement (1)

-0.001 -0.01 18,300


0.01 0.01 18,347

Credits attempted -0.35*** -0.42*** 23,707Credits earned -0.42*** -0.62*** 23,707Regents attempted -0.01 -0.04 23,707


Note: * p<0.05, ** p<0.01, *** p<0.001 Each cell represents a coefficient estimate from a separate regression, where the indicated outcome variable is used. Models include student-level covariates and cohort dummies.

33

LIMITATIONS


Agnostic about the motivation behind students’ rankings of schools; have yet to investigate mechanisms

Outcomes: smaller sample with School Survey data may represent restricted range of engagement

Random assignment prevalent in limited unscreened schools, but different assignment mechanisms require different analytic approach

34

NEXT STEPS


Expand analyses by school type

Examine results for patterns across school type

35

Sean P. Corcoran Lori Nathanson James Kemple New York University INVALSI – Rome, Italy – October 2012 T HE I MPACT OF H IGH S CHOOL C HOICE ON M EDIATORS.

Documents

s chool c hoice preliminary

h igh s chool c hoice

high school choice

engagement preliminary

value of school

matched school

s chool multiple dimensions

exams preliminary