1 Assessing the Effects of Advanced Placement Policies Lauren Linzmeier May 2012 Mathematical Methods in the Social Sciences Northwestern University Advisor: Professor Kirabo Jackson Abstract This thesis investigates whether Advanced Placement policies achieve their goals of increasing AP enrollment and improving college outcomes. Using aggregate longitudinal data on AP exam taking, college enrollment, and college graduation rates from all 50 states, I find that very few types of AP policies work to increase AP exam taking or improve college outcomes. However, two of the most costly AP policies, financial incentives for exam performance and national AP incentive grants, do seem to increase AP exam taking. Several sources of bias in the college graduation rate data prevent me from drawing decisive conclusions regarding the effect of AP policies on college graduation rates, but I find little evidence to suggest that any AP policies work to increase college persistence. Acknowledgements This thesis would not have been possible without the support of several outstanding faculty members. I would like to thank Professor Jackson for serving as my advisor and helping me through the conceptual and analytical portions of this project. Professor Canay provided invaluable econometric guidance, and Professor Witte sacrificed countless hours to help me see this thesis to its conclusion. I would also like to thank Professor Figlio for his guidance, inspiration, and mentoring throughout my time at Northwestern.
47
Embed
Assessing the Effects of Advanced Placement Policiesmmss.wcas.northwestern.edu/thesis/articles/get/793/Linzmeier2012.pdf · ! 1! Assessing the Effects of Advanced Placement Policies
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Assessing the Effects of Advanced Placement Policies
Lauren Linzmeier
May 2012
Mathematical Methods in the Social Sciences Northwestern University
Advisor: Professor Kirabo Jackson
Abstract This thesis investigates whether Advanced Placement policies achieve their goals of increasing AP enrollment and improving college outcomes. Using aggregate longitudinal data on AP exam taking, college enrollment, and college graduation rates from all 50 states, I find that very few types of AP policies work to increase AP exam taking or improve college outcomes. However, two of the most costly AP policies, financial incentives for exam performance and national AP incentive grants, do seem to increase AP exam taking. Several sources of bias in the college graduation rate data prevent me from drawing decisive conclusions regarding the effect of AP policies on college graduation rates, but I find little evidence to suggest that any AP policies work to increase college persistence.
Acknowledgements This thesis would not have been possible without the support of several outstanding faculty members. I would like to thank Professor Jackson for serving as my advisor and helping me through the conceptual and analytical portions of this project. Professor Canay provided invaluable econometric guidance, and Professor Witte sacrificed countless hours to help me see this thesis to its conclusion. I would also like to thank Professor Figlio for his guidance, inspiration, and mentoring throughout my time at Northwestern.
2
I. INTRODUCTION
Expanding access to advanced placement programs would provide more disadvantaged high school students the opportunity to take challenging courses so that they will enter college or the global marketplace ready to excel. — U.S. Secretary of Education Margaret Spellings, February 2006
Since its inception in 1955, the Advanced Placement (AP) program has captured the
attention of policymakers who believe exposing students to rigorous Advanced Placement
classes will increase high school achievement and college success. State education policies
since the 1980s have focused on expanding access to AP coursework, especially among
disadvantaged students, and improving AP course quality and exam performance. In the 1980s,
some states began mandating AP offerings, tied accountability incentives to AP participation
rates, and subsidized student test fees. More states followed suit in the 1990s and 2000s, and
President Bush in his 2006 State of the Union Address pledged to expand the Advanced
Placement program as a key component of his American Competitiveness Initiative.
Various state and national polices aim to increase participation in the Advanced
Placement program in the hopes that the program will prepare students for advanced coursework
and increase college success. While some of these policies require no additional funding, others
are quite expensive. In 2011, the United States Department of Education spent over $23 million
funding its Advanced Placement Test Fee Program, and yearly appropriations for the National
AP Incentive Program have ranged between $18 and $31 million. The high costs of some of
these policies and programs beg the question: which, if any, effectively increase participation on
the AP program? More importantly, do any of these policies work through the AP program to
better prepare students for college and increase college enrollment or graduation rates?
This thesis investigates whether cohorts of high school students in states subject to AP
policies increase their AP exam taking, perform better on AP exams, and ultimately have higher
3
college enrollment and graduation rates using a differences-in-differences approach. Using
aggregate data from 1994-2009, I find that having financial incentive programs funded either by
the state or national government increases AP exam taking and AP exam performance, indicating
that AP polices can successfully increase AP participation, but I find little evidence these AP
policies successfully increase college enrollment and graduation rates.
The paper proceeds as follows. The following section (II) reviews prior literature on the
effects of the AP program and AP policies on college outcomes. Section III provides some
background on the AP program and state and national AP policies. Section IV describes the data,
while section V presents the empirical approach and results. Section VI summarizes the key
findings and discusses their implications.
II. LITERATURE REVIEW
Due to the popularity of the AP program, many researchers have been interested in the
effects of AP participation on high school and college outcomes. Some literature links success in
the AP program with improved college outcomes. Even after controlling for high school rank,
SAT score, and pre-college criteria, students with advanced placement credit earn higher first
semester college grades than students without advanced placement credit (Keng & Dodd, 2008;
Hargrove, Godin, & Dodd, 2008; Scott, Tolson, & Lee 2010). Taking AP courses also increases
the likelihood that a student will graduate from a four-year college or university within six years
(McCauley, 2007; Hargrove, Godin, & Dodd 2008) and earn a graduate degree (Bleske-Rechek,
Lubinksi, & Benbow 2004).
4
However, a few researchers question the benefits of AP participation. Klopstein &
Thomas (2009) find that after controlling for a student’s non-AP curricular experience, AP
participation does not predict college grades or persistence. Geiser & Santelices (2004) find that
while satisfactory performance on AP exams predicts improved college outcomes, merely taking
an AP course has no effect on college performance. This is likely because students who do not
feel prepared to take the AP exam have not mastered the course material or significantly
improved their academic abilities in the course. Hallett & Venegas (2011) suggest that
expanding access to the AP program is not sufficient to improve college outcomes of
disadvantaged students if students do not perform well in the courses. Johnson (2004) argues
that allowing promising high school students to use Pell grants to take actual college courses
would be a more cost-effective way to increase college level opportunities for low-income
students than AP incentive policies.
Very few studies have looked at the effects of state Advanced Placement policies
directly. Of the two recent studies that have, each has adopted a different approach than this
analysis. Jackson (2010) uses high-school level data, urbanicity data, college enrollment data,
and a listing of schools participating in the Texas Advanced Placement Incentive Program by
year in order to evaluate the effects of the Texas Advanced Placement Incentive Program. He
finds that the program, which pays students and teachers for passing grades on AP exams, results
in more AP course and exam taking, increases SAT/ACT scores, and promotes college
matriculation. His more recent NBER working paper analyzes longer-term outcomes. In this
paper, Jackson (2010) finds that the incentive program increases college GPAs, increases college
persistence, and increases college graduation rates for black and Hispanic students.
5
By contrast, another study considers AP policies across all states. Jeong (2009) analyzes
microdata from the National Educational Longitudinal Study of 2002 to determine whether test
fee waivers or performance-based incentives increase AP participation and success. He
concludes that test fee subsidies successfully increase the probability that a student will take an
AP exam while performance-based incentives have little or no effect. This seems to conflict
with Jackson’s findings that performance-based incentives are quite effective. Because Jeong
(2009) only has data points from 2002, he is unable to control for state fixed effects, though he
does control for regional effects. Moreover, he only investigates the effect of fee waivers and
performance incentives on AP exam taking and performance in four popular AP subjects rather
than long-term college outcomes.
The analysis in this paper uses aggregate statistics on AP performance by state for 1994-
2009. Analysis of this panel data avoids some the problems Jeong (2009) encountered.
Dramatically different trends between neighboring states with similar policy profiles suggest that
state level fixed effects are necessary for valid identification, and this analysis controls for state
fixed effects. Moreover, this data set allows for an examination of more than just test fee waivers
and performance-based incentives. In fact, this analysis considers ten different types of
Advanced Placement policies. Finally, looking at data on college enrollment rates and college
graduation rates provides useful insight about the long-term effects of AP policies.
III. BACKGROUND
Description of the Advanced Placement Program
Educators widely praise the Advanced Placement (AP) program for giving students the
opportunity to take advanced coursework. The Advanced Placement program has become
6
increasingly popular over the past several decades and consists of 34 courses and corresponding
exams. The College Board, a not-for-profit organization, regulates the Advanced Placement
program by publishing course descriptions, approving audits from AP instructors, and creating
and scoring AP exams. Though it is possible for students to prepare for an AP exam on their
own, most students take an AP course at their local high school prior to taking the corresponding
exam in May. The College Board charges $87 for each exam but offers a partial test fee waiver
for low-income students.
Overview of State Policies1
In response to the perceived benefits of Advanced Placement, some states have
implemented a wide range of policies to support the program’s growth and encourage student
participation. These policies can be roughly divided between those that target schools and
districts and those that appeal to students directly. Policies targeting schools and districts include
mandating the offering of AP courses, financial exam performance-based incentives,
accountability incentives, offering funds for teacher training, mandating teacher training, and
policies to encourage collaboration between colleges and universities and K-12 districts. Policies
targeting students include AP exam fee subsidies and guaranteeing credit at state colleges for
qualifying AP scores.
The policies targeting districts and schools mandate the expansion of the AP program,
offer financial assistance, or create financial or accountability incentives. For instance,
Arkansas, Indiana, West Virginia, and South Carolina mandate that all high schools offer a
minimum number of AP courses in an effort to expand access to low-income and minority
1 Based on the Education Commission of the State’s State Notes Advanced Placement Report 2 See the Elementary and Secondary Education Act of 1965, as amended, Title I, Part G, Sec. 1704-05
7
students whose schools tend to offer fewer AP courses. Other states offer financial assistance for
AP development at low-income and small schools. This assistance usually consists of grants to
purchase textbooks and classroom materials. Additionally, some states offer rewards to schools
with high numbers of students who complete AP courses and/or receive at least a 3 on the exam.
Some states also hold schools accountable for their AP performance. Colorado and Michigan tie
AP offerings to accreditation while fifteen other states include AP participation as part of the
district, school, or state’s report cards and accountability ratings. Meanwhile, some states
require schools and districts to publish statistics on AP performance.
States have also enacted measures to improve the quality of the AP instruction. Eleven
states have adopted policies to encourage collaboration between higher education institutions and
K-12 schools in offering AP courses. Twenty-seven states make funds available for AP teachers
to attend summer institutes and other AP or pre-training programs while five states mandate that
AP teachers complete AP training.
The remaining policies appeal to students directly. In addition to the fee reduction the
College Board makes available to free and reduced price lunch students, states use additional
U.S. Department of Education funds to subsidize AP exams. In some states, these subsidies are
available to low-income students only while other states allow higher income students who are
taking multiple AP exams to benefit from the funds. Moreover, the amount of the subsidy varies
between states. Finally, some states guarantee college credit at in-state institutions to any student
who scores a three or higher on the AP exam for a subject.
Overview of National Policies
The national government has also implemented policies to support the expansion of the
Advanced Placement program. These policies generally involve grants to individual states. For
8
instance, Congress authorized the federal AP fee assistance program in the 1992 Higher
Education Act. However, the program was not funded until the 1998 fiscal year when thirty-two
states received federal monies. This program continued until 2007 when it was replaced by the
Advanced Placement Test Fee Program2 which awards grants to state agencies for the purpose of
reimbursing low-income students enrolled in AP courses who take AP exams. This program
granted awards starting in 2008 and appropriations have increased over time from $11,507,910
in 2008 to $23,343,981 in 2011. Meanwhile, the Advanced Placement Incentive Program
provides three year competitive grants to State educational agencies, local educational entities, or
national nonprofit educational entities to increase the participation of low-income students in the
AP program. Entities must use the grants to fund professional development for teachers, develop
curriculums, purchase books and supplies, or “develop, enhance, or expand” AP or pre-AP
courses. Preference is given to entities that demonstrate a “pervasive need for access to advance
placement programs,” “assure the availability of matching funds,” and “provides for the
involvement of businesses and community organizations.” This program has granted new awards
or continued awards every year since 2001 with yearly appropriations ranging between
$18,410,799 and $31,573,442.
Two Dimensions of Policy Variation: Content and Timing
This analysis exploits variation in the timing and content of Advanced Placement policies.
States without particular policies in particular years serve as controls. Among states who have
implemented AP policies, there is substantial variation in the timing of the policies and their
content. See Appendix A for a summary of variation in the timing and content of AP policies.
2 See the Elementary and Secondary Education Act of 1965, as amended, Title I, Part G, Sec. 1704-05
9
Coding State Policies
While there are subtle variations even among policies of a given type, drawing any
conclusions requires roughly categorizing AP policies. For the purposes of this analysis, I code a
policy as one of ten policy types: (1) equipment or instructional grants to establish AP courses,
(2) mandating AP course offerings, (3) accountability incentives, (4) subsidies for AP exam fees,
(5) mandating or funding teacher training and professional development, (6) mandating credit for
passing AP exam scores at in-state institutions, (7) collaboration between K-12 and higher
education institutions, and (8) financial incentives for AP exam performance. I code the two
national policies as (9) test fee reductions funded by the U.S. Department of Education or (10)
AP incentive grants. States in a particular year are coded as having a particular policy if the
policy was in effect during that exam year or if they received grant money under one of the two
national policies in that exam year.3 For example, if a particular policy began in 2001-2002, it is
coded as beginning in 2002 since we would expect the policy to have an effect on the 2002 AP
exam period in May, not during the AP exam period of the previous school year when the policy
was not in effect.
IV. DATA
This analysis uses four datasets in addition to the dates of implementation for various AP
policies. The first dataset consists of Advanced Placement data. This panel of data, constructed
from aggregate totals in publically available College Board state summary reports, gives the
number of students who took AP exams and the total number of AP exams taken for every year
3 Determining the date of implementation for these policies is sometimes quite difficult. The Education Commission of the States database on AP policies was used whenever possible. When the ECS sources provide no date, Lexus-Nexus or U.S. Department of Education data was used.
10
between 1994 and 2009, inclusive, by state. Note that these data cannot be used to determine
how many students took AP courses but did not take any AP exams. Consequently, this data can
only be used to investigate whether particular policies increased AP exam taking, not whether
they increased the number of students enrolling in Advanced Placement. However, given that
prior research finds that only students who opt to take the exam benefit from AP participation
(Geiser & Santelices 2004), this exam taking data is of the most interest.
The second dataset from the Common Core of Data and the National Center for Education
Statistics consists of educational statistics for state/year cohorts between 1994-2009. These data
give the number of 11th and 12th grade students, the number of black students in 12th grade, the
number of white students in 12th grade, the number of Hispanic students in 12th grade, average
expenditure per student, and number of students eligible for free lunch. I use these totals to
construct controls for each cohort and variables that give the number of AP students and AP
exams per 1,000 high school juniors and seniors for 1994-2009.
The third dataset provides information on AP exam performance. The National Center for
Higher Education Management Systems constructed a dataset from the College Board state
summary reports which gives the number of AP exams scores at 3 or above per 1,000 high
school juniors and seniors by state for 1998-2003. Since it may be that some AP policies
improve AP course quality (and thus increase the number of AP exams scored three or higher)
but do not increase the number of students choosing to take an AP exam, this dataset will capture
different effects than the first AP dataset.
Finally, a fourth dataset consists of three different college outcomes by state and year: the
enrollment rate, the three-year graduation rate for associate’s students, and the six-year
11
graduation rate from bachelor’s students.4 The enrollment rate for state X is defined as the
number of first-time freshman who graduated from high school in the past year in state X
enrolled anywhere in the U.S. This reflects the college going rate of graduating high school
seniors in a particular state and year. The three-year graduate rate of associate’s students is
defined as the percentage of first-time full-time associate degree-seeking students earning any
formal award (certificate or associates degree) within three year at a Title IV degree-granting
institutions between 1997 and 2009 inclusive. Thus, these data correspond to the 1994-2006
high school senior cohorts in each state, assuming that all the first-time associate’s degree
seeking students enrolled in the fall immediately after their high school graduation. The six-year
graduation rate of bachelor’s students is defined as the percentage of first-time full-time
bachelor’s degree-seeking students earning any formal award (certificate, associate, or bachelors
degree) within six years at a Title IV degree-granting institution between 1997 and 2009,
inclusive. Thus, these data correspond to 1991-2003 high school senior cohorts in each state,
assuming that all the first-time bachelor’s seeking students enrolled in the fall immediately after
their high school graduation.
In the analysis that follows, I attempt to estimate the effect of an AP policy on the
eventual college graduation rate of a cohort of high school seniors subject to that policy. To do
this, I look at the graduation rates of the college cohorts in the state and year the fall after that
senior cohort graduated. There are three obvious problems with this approach. First of all,
neither of these graduation rates account for transfers across institutions. If one state has a larger
percentage of students transferring than another, all else equal, this state will have a lower
graduation rate in this data. Moreover, this data may suffer from systematic reporting bias.
4 The National Center for Higher Education Management Systems compiled this data from the National Center for Education Statistics Integrated Postsecondary Education Data System Graduation Rate Survey.
12
Anywhere between 33% and 100% of the institutions in a particular state responded to the
IPEDS Graduation Rate Survey. It may be that institutions with the highest graduation rates are
those most likely to respond to the survey. In this case, the data may not accurately reflect actual
college graduation rates. Finally, using this graduation rate data to evaluate the effects of AP
policies on the eventual graduation rates of particular cohorts relies on two strong assumptions. It
assumes that seniors enroll immediately in the fall after their high school graduation and attend
an institution in the same state as which they attended high school. Obviously, many students
take a gap year between high school and college or choose an out of state institution. I include
estimates for the effect on graduation rates for illustrative purposes, but due to the problems
outlined above, these results should be interpreted cautiously.
Before presenting my empirical results, I discuss possible trending in my AP and college
outcomes data.
Trends in the National AP Data
Nationally, the number of students taking at least one AP course has increased
exponentially since 1973 when the College Board began reporting enrollment data. The number
of students who took at least one AP exam ranges from 53,902 in 1973 to 1,802,144 in 2010. The
total number of schools with students who took AP exams has also increased every year from
3,114 in 1973 to 16,699 in 2010. The number of AP exams scored 3 or higher per 1,000 high
school juniors and seniors has also increased every year between 1997 and 2007, the years for
which data is available.
Trends in National College Enrollment and Graduation Rate Data
13
College enrollment rates have been generally increasing over time as shown in figure 2.
In 1992, the college enrollment rate was 52%. The rate increased until 2000 and then slipped
slightly before increasing monotonically to 62.5% in 2008. Graduation rates have fluctuated over
time nationally, though they seem to be generally increasing as shown in figure 3. The 1992 and
1993 high school senior cohorts had the lowest graduation rates during the thirteen-year period
with 52% of the cohort earning a bachelor’s degree within six years. Very few states had any AP
policies during this period. Whether due to the expansion of the AP program or other factors,
graduation rates increased in the following period. They peaked at 56.4% for the 2000 cohort,
but declined for each of the three subsequent cohorts.
V. EMPIRICAL RESULTS
I. Effect of an AP Policy
The first part of this empirical analysis compares states with no Advanced Placement policies
to states that have at least one AP policy in order to determine whether Advanced Placement
policies successfully increase the number of students who participate in the AP program,
improve AP exam performance, and ultimately increase college going or college graduation
rates. A state is considered to have an AP policy in a given exam year if they have any of the
eight state AP policies or receive grant money under either national AP policy.
Before presenting the results of more rigorous regression analysis, I present a few illustrative
figures. Figure 1 examines the relationship between number of states with AP Policies and
number of AP students. First, I plot the number of students who took at least one AP test per
14
1,000 high school juniors and seniors for 1991-20035. Below this, I plot the number of states
with any AP policy in those years. While the number of states with AP policies fluctuates
somewhat, over the entire period, the number increases from 7 to 42. Meanwhile, the number of
AP students per 1,000 high school juniors and seniors increases monotonically. At first glance,
this figure suggests that AP policies may increase the number of students who take AP exams.
However, it is notable that the number of students who take AP exams does not decrease when
the number of states with AP policies decreases in 2001 and 2002. It could be that AP exam
taking is increasing over this period for reasons other than AP policies.
In figure 2, I plot the number of states with AP policies against the college enrollment rate. A
drop in the number of states with AP policies corresponds to a drop in the college enrollment rate
between 2000 and 2002, but in all other years in the 1992-2008 period, both are increasing
monotonically. This figure associates greater AP policy implementation with increases in the
college enrollment rate. Of course, rigorous regression analysis with some identifying
assumptions will be needed to test a casual claim.
In figure 3, I plot the number of states with AP policies against the college graduation rate
nationally. During the period 1991-2000, both the number of states with AP policies and the
college graduation rate is increasing. Then in 2001-2002, both drop. In 2003, the number of
states with AP policies increases again, but the college graduation rate continues to fall. Once
again, it is hard to draw a definitive conclusion regarding the relationship between the two
variables, but overall, both seem to be increasing during this time period.
Finally, I break down the data by state. For figure 4, I compute the change in the college
enrollment rate from 1996 to 2008 for each state as well as the number of years during the period
that each state had at least one AP policy in effect. For figure 5, I compute the change in the 5 I present the 1991-‐2003 time period so that this figure can be directly compared with figure 3.
15
college graduation rate from 1991 to 2003 for each state as well as the number of years during
the period that each state had at least one AP policy in effect. Figures 4 and 5 present
scatterplots of these relationships. If AP policies increase college enrollment or graduation rates,
there should be evidence of a positive correlation. Figure 4 shows a positive correlation, which is
consistent with the idea that AP policies increase college enrollment. By contrast, figure 5
shows no correlation; the best fit line is horizontal. This suggests that AP policies do not
increase college graduation rates.
Of course, these figures make no effort to control for differences between states with and
without AP policies along other dimensions. Perhaps states who pass AP policies tend to be
states with student populations more or less amiable to the AP program or states with student
populations more or less able to take AP exams and perform well. In order to control for
differences along other dimensions, I estimate two regression model specifications using OLS:
The dependent variable is 𝑦!" denotes an AP outcome or college outcome for state s in year t.
The AP outcomes I investigate are APstudents, APexams, and exams3above. APstudents denotes
the number of students who took at least one AP exam per 1,000 high school juniors and seniors.
APexams denotes the total number of AP exams taken per 1,000 high school juniors and seniors.
Exam3above denotes the number AP exams scored 3 or above per 1,000 of high school juniors
and seniors. Additionally, I investigate several college outcomes: the percentage of graduating
high school seniors that enroll immediately in college the following fall (enrollrate), the
percentage of first-time associate’s degree-seeking students earning a degree within three years
(gradrate3year), and the percentage of first-time, full-time bachelor’s degree-seeking students
16
earning a degree within six years (gradrate6year). These models include one explanatory
variable: 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!". This explanatory variable is equal to 1 if state s had any one of eight
state AP policies or received grants under either national AP policy in year t and 0 otherwise.6 In
all specifications, I include the following controls for the year/state cohort: percent black, percent
white, percent Hispanic, percent eligible for free lunch, and average educational expenditures per
student. In all these models, 𝜀!" is the idiosyncratic error for state s in year t.
In the first specification, I include year fixed effects (𝛼!) to control for factors affecting AP
or college outcomes across all states in a particular year. These fixed affects should capture the
general trending upward in the AP outcomes data as well as more complicated trends in the
college outcomes data. Then in the second specification, I also add state fixed effects (𝛾!) to
control for differences between states during the time period. There are several reasons to
include a state fixed effect. First of all, states have different high school student populations that
may vary by ability, initial willingness to participate in AP, and familiarity with the AP program.
States also have different teacher populations that may vary in their willingness to teach AP
courses, ability to teach AP courses, and enthusiasm for the AP program. This specification
assumes that all of the factors captured by the state fixed effect are roughly constant over the
time period of analysis.
Table 1 displays the estimates for these two regression models with six different dependent
variables: APstudents, APexams, exams3above, enrollrate, gradrate3year, and gradrate6year.
I code 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!" as 1 if the policy was in effect when students the year t and state s cohort
6 A state is considered to have an state AP policy if it has any one of the following eight state AP policies: equipment or instructional grants to schools or districts, mandates for AP course offerings, accountability incentives, state AP test fee subsidies (independent of those funding by national test fee reduction grants from the U.S. Department of Education), mandating or providing funds for AP teacher training or professional development, financial incentives for AP exam performance, guaranteeing college credit at public in-‐state Institutions for passing AP exam scores, and collaboration between K-‐12 and higher education systems regarding the AP program.
17
were seniors. Then, I estimate outcomes for that year/state cohort of seniors. Table 1 shows that
point estimates for the effect of 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦 is positive for all three AP outcome variables in
specification 1, and 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦 has a significant effect for both APexams and exams3above.
There seems to be no effect on any college outcomes. Including state fixed effects generally
decreases the magnitude of the point estimates, but now the point estimates for APstudents and
APexams are positive and significant at the 5% level. The estimate for exams3above is positive
but insignificant. There is a surprisingly large and significant effect for the college enrollment
rate, but there is no significant effect for college graduation rates. These point estimates imply
that all else equal, states with at least one AP policy have 8.1 more AP students and 15.8 more
total AP exams per 1,000 high school juniors and seniors and college enrollment rates 2.5%
higher than states without.
In order to test the validity of these results, I run a falsification test. I estimate effects for
cohorts two years before the state’s first AP policy. (I replace 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!" with
𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!! in my regression specifications.) Since this is two years before the real
treatment, I should detect no effects as long as there is no trending in the data that state and year
fixed effects fail to capture. The table below table 2 shows that the data passes this falsification
test. There are no significant effects on AP outcomes or college outcomes in either specification
(1) or (2).
Additionally, I probe the robustness of the finding that AP policies increase AP outcomes
and possibly college enrollment by dropping particular years from the regressions. The size and
significance of the resulting estimates are sensitive to what years are included, though the
estimates are always positive. For example, dropping data from 1994-1997 dramatically reduces
the size of the estimates and makes them insignificant. This is likely because one or two salient
18
AP policies drives the result that AP policies increases AP outcomes and enrollment, and this
particular policy was turning on or off in several states during this 1994-1997 period.7
Next, I move beyond a simple pre/post analysis to investigate the cumulative effects of these
policies. To specification 2, which already includes state and year fixed effects, I add four pre-
policy explanatory variables
(𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!! , 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!! , 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!! , 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!!) as well 3 post-policy
explanatory variables (𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!! , 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!! , 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!!) to trace the number of
years before or after the state implemented its first AP policy. The base case is five or more
years before the policy. For example, the coefficient 21.9 on 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦!"!! in column (1) means
that a cohort of high school juniors and seniors subject to an AP policy in its third year had on
average 21.9 more AP students than five years before an AP policy, all else equal.
I plot the point estimates for effects on the number of AP students and the college enrollment
rate over time in a graph below table 2. Since none of the coefficients are significant, one should
be wary of drawing sharp conclusions from table 2 or the corresponding graph. However, the
positive and increasing point estimates for third and fourth year of the AP policy suggest that
states may see an increase in number of AP students and college enrollment rates a few years
after AP policy implementation. This is consistent with policy lags, which in the context of state
AP policies seem quite likely. This graph also shows that increases in college enrollment rates
generally correspond with increases in the number of AP students. This suggests some effect on
enrollment rates after several years of AP policies, but since the estimates are not significant,
these results do not provide strong evidence for such a relationship.
7 Later in the paper when I break down the results by particular AP policies, I find evidence that financial incentives and national AP incentive grants drive this result. Of the only five states that have ever implemented financial incentives, 3 began in the 1994-‐1997 time period (Arkansas (1995), Texas (1996), and Oklahoma (1997)). The other two began shortly after (Florida (1998) and Colorado (1999)).
19
II. Effects of Particular State AP Policies
Up until this point, I have lumped all AP policies into a single category. Of course, not all
policies are created equal. Some focus on expanding access to AP; some focus on increasing AP
exam taking, and the remaining policies focus on improving AP quality. For example, national
AP incentive grant recipients must use the funds to increase participation of low-income students
in the AP program, expanding access. Some states provide AP exam test fee waivers to
encourage AP students to take exams at the conclusion of their AP courses, hoping that this will
cause students to work harder to learn the material and earn college credit. Finally, states that
offer funds for teacher training and professional development strive to better equip AP
instructors to offer quality courses.
Various types of incentives may make some policies more effective than others. In the table
below, I list various aspects of AP policies and their corresponding goals. In the analysis that
follows, I attempt to evaluate how well each of the ten types of policies are at achieving the three
AP-related goals of (1) expanding access, (2) increasing AP exam taking, and (3) increase AP
exam performance. In addition, I assess how well each type of policy works to increase college
enrollment and graduation rates.
20
Goal Policies Targeting Schools and Teachers
Policies Targeting Students
Primary AP Outcome of
Interest Encourage
Participation Broadly/Expand
Access (1)
- Reward schools with higher AP enrollments - Include AP enrollment percentages as an accreditation factor - Equipment grants to schools - Require schools to publish AP enrollment statistics - Equipment grants to low-income schools - Mandate course offerings
- Offer weighted grades for AP courses - Offer distance learning and online AP courses - Require parental and student notification of AP offerings
APStudents
Increase AP Exam Taking
(2)
- Require schools to publish rates of exam taking in accountability reports
- Mandate that students take exams to earn course credit - Test fee waivers
APExams
Increase AP Exam Success
(3)
- Require schools to publish AP exam results in accountability reports - Provide funds for teacher training - Mandate teacher training - Financial incentives for teachers - Collaboration between K-12 and higher education
- Guarantee college credit for passing scores at state institutions - Make state scholarships dependent on AP exam performance
Exams3Above
In order to investigate the effects of particular AP policies, I estimate a model very similar to
the specification 2 from part I, except I replace 𝑎𝑛𝑦𝐴𝑃𝑝𝑜𝑙𝑖𝑐𝑦 with 8 variables for the eight types
of state AP policies and 2 variables for the two types of national AP policies. As before, I
include a state and year fixed effects, the model is as follows:
Geiser, S., & Santelices, V. (2004). The role of Advanced Placement and honors courses
in college admissions (Research & Occasional Paper Series: CSHE.4.04).
Berkeley: University of California.
Hallett, R, E,. & Venegas, K. M. (2011). Is Increased Access Enough? Advanced
Placement Courses, Quality, and Success in Low-Income Urban Schools. Journal
for the Education of the Gifted, 34(3), 468-487.
Jackson, Kirabo C. (2010). A Little Now for a Lot Later: A Look at a Texas Advanced
Placement Incentive Program. The Journal of Human Resources. 45(3), 591.
Jackson, Kirabo C. (2010). Effects of an Incentive-Based High School Intervention on
32
College Outcomes. National Bureau for Economic Research Working Paper
15722. http://www.nber.org/papers/w15722
Jeong, D.W. (2009). Student Participation and Performance on Advanced
Placement Exams: Do State-Sponsored Incentives Make a
Difference? Educational Evaluation and Policy Analysis. 31(4), 346-366.
Johnson, Kirk. (2004). Expanding opportunity for Low-Income High School Students:
Pell Grants vs. Advanced Placement Classes. The Heritage Foundation.
Keng, L. & Dodd, B. G. (2008). A comparison of college performances of AP® and
non-AP student groups in 10 subject areas (College Board research report no.
2008-7). New York, NY: The College Board.
Klopfenstein, K., & Thomas, M. K. (2009). The link between advanced placement
experience and early college success. Southern Economic Journal, 75(3), 873-
891.
McCauley, David. (2007) The Impact of Advanced Placement and Dual Enrollment
Programs on College Graduation. Applied Research Projects, Texas State
University-San Marcos. Paper 206.
http://ecommons.txstate.edu/arp/206
Scott, P., Tolson, H., & Yi-Hsuan, L. (2010). Assessment of Advanced
Placement Participation and University Academic Success in the First Semester:
Controlling for Selected High School Academic Abilities. Journal of College
Admission, (208), 26-30.
-----. (2010). Number of Eleventh and Twelve Grade Students by State, Number of
Black, White, and Hispanic Students, Number of Free Price Lunch Students,
Average Expenditure Per Student. Common Core of Data. National Center for
Education Statistics.
-----. (2010). Number of AP Exam Scores at 3 or Above Per 1,000 High School Juniors
and Seniors: Advanced Placement Data 1997-2003, 2005, and 2007.
National Center for Higher Education Management Systems.
-----. (2009). College-Going Rates of High School Graduates – Directly from High
School. National Center for Higher Education Management Systems.
-----. (2009). Progress and Completion: Graduation Rates Data 1997-2009. National
33
Center for Higher Education Management Systems.
-----. (2011). State Notes: Advanced Placement. Education Commission of the States.
Retrieved November 1, 2011 from http://mb2.ecs.org/reports/Report.aspx?id=922
-----. (2011). U.S. Department of Education, Office of Academic Improvement and
Teacher Quality Programs: Advanced Placement Incentive Program Grants.
Retrieved November 1, 2011 from http://www2.ed.gov/programs/apincent/index.html
-----. (2011). U.S. Department of Education, Office of Academic Improvement and
Teacher Quality Programs: Advanced Placement Test Fee Program.
34
FIGURES AND TABLES:
0 5 10 15 20 25 30 35 40 45 50
0
20
40
60
80
100
120
140
160
180
1991 1993 1995 1997 1999 2001 2003
Figure 1: Relationship between Number of States with AP Policies and Number of AP
Students
Number of Students Per 1,000 HS Juniors and Seniors that Took at Least One AP Exam
Number of States with AP Policies
52 53 54 55 56 57 58 59 60 61 62 63
0
10
20
30
40
50
60
1990 1995 2000 2005 2010
Figure 2: Relationship Between Number of States with AP Policies and the College
Enrollment Rate
Number of States with At Least One AP Policy
College Enrollment Rate
35
0 5 10 15 20 25 30 35 40 45 50
51.5 52
52.5 53
53.5 54
54.5 55
55.5 56
56.5 57
1991 1993 1995 1997 1999 2001 2003 Cohort Year
Figure 3: Relationship Between Number of States with AP Policies and the National Six-‐
Year Graduation Rate
Graduation Rate (Percent)
Number of States with AP Policies
-‐15
-‐10
-‐5
0
5
10
15
20
25
0 2 4 6 8 10 12 14 16
Change in the College Enrollment Rate
Years with an AP Policy
Figure 4: Change in Six-‐Year Graduation Rate vs. Years with AP Policies (1996-‐2008)
36
-‐8
-‐6
-‐4
-‐2
0
2
4
6
8
10
12
14
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Figure 5: Change in Six-‐Year Graduation Rate vs. Years with AP Policies (1991-‐2003)
37
Table 1: Effect of Any AP Policy on AP and College Outcomes (1994-2009)
Falsification Test (1994-2009)
Specification (Row 1): Year Fixed Effects Only Specification (Row 2): Year Fixed Effects and State Fixed Effects Robust standard errors in brackets * significant at 5%; ** significant at 1%
Specification:
(1) AP Students Per 1,000 HS Juniors and Seniors
(2) AP Exams Per 1,000 HS
Juniors and Seniors
(3) AP Exams Passed Per 1,000 HS
Juniors and Seniors
(4) Percentage of High School Graduates Enrolling in College
Immediately After HS
Graduation
(5) Three-‐Year College
Graduation Rate of
Associate’s Degree Seeking
Students
(6) Six-‐Year College
Graduation Rate of Bachelor’s Degree Seeking Students
(1) Any AP Policy 16.054 [8.240]
29.882* [13.751]
0.830** [0.286]
0.827 [1.852]
-‐1.574 [1.692]
0.528 [1.302]
(2) Any AP Policy 8.092*
[3.862] 15.781* [7.223]
3.683 [2.230]
2.487** [0.880]
0.130 [0.858]
0.026 [0.321]
Observations 716 716 385 337 538 444
Specification:
(1) AP Students Per 1,000 HS Juniors and Seniors
(2) AP Exams Per 1,000 HS
Juniors and Seniors
(3) AP Exams Passed Per 1,000 HS
Juniors and Seniors
(4) Percentage of High School Graduates Enrolling in College
Immediately After HS
Graduation
(5) Three-‐Year College
Graduation Rate of
Associate’s Degree Seeking
Students
(6) Six-‐Year College
Graduation Rate of Bachelor’s Degree Seeking Students
(1) Two Years Before Any AP
Policy
-‐4.330 [4.235]
-‐8.818 [7.338]
-‐4.033 [7.717]
-‐0.348 [1.468]
-‐1.506 [1.625]
-‐1.053 [0.776]
(2) Two Years Before Any AP
Policy
-‐3.032 [2.438]
-‐5.660 [4.592]
-‐4.845 [4.886]
0.571 [0.974]
-‐0.847 [15.682]
-‐0.128 [0.497]
Observations 716 716 385 337 538 444
38
Table 2: Effect of Any AP Policy on AP and College Outcomes Over Time (1994-2009) (Specification 2: Year Fixed Effects and State Fixed Effects)
(1) AP Students Per 1,000 HS Juniors and Seniors
(2) AP Exams Per
1,000 HS Juniors and Seniors
(3) Percentage of High School
Graduates Enrolling in College
Immediately After HS
Graduation
(4) Three-‐Year College
Graduation Rate of
Associate’s Degree Seeking Students
(5) Six-‐Year College Graduation Rate of Bachelor’s
Degree Seeking Students
Four Years Before Any AP Policy
-‐1.634 [14.160]
-‐1.469 [7.813]
2.367 [1.655]
-‐6.902 [4.665]
0.889 [1.020]
Three Years Before
Any AP Policy 1.984
[16.129] -‐0.200 [8.792]
-‐1.518 [1.894]
-‐8.854 [4.663]
-‐0.244 1.134]
Two Years Before Any AP Policy
5.012 [18.592]
1.32 [10.214]
2.282 [1.925]
-‐8.065 [4.841]
-‐0.026 [1.213]
One Year Before Any AP Policy
8.197 [21.440]
2.766 11.777]
-‐0.956 [2.236]
-‐8.037 [5.304]
-‐0.021 [1.286]
First Year with An AP Policy
13.87 [22.952]
6.019 [12.442]
2.777 [2.354]
-‐6.972 [5.755]
-‐0.283 [1.266]
Second Year with An AP Policy
10.966 [26.368]
4.301 [14.341]
2.472 [2.473]
-‐7.862 [5.146]
-‐0.550 [1.245]
Third Year with An AP Policy
21.883 [27.078]
9.395 [14.791]
0.988 [2.907]
-‐7.421 [5.379]
-‐0.207 [1.265]
Four+ Years with An AP Policy
45.931 [30.697]
21.919 [16.471]
3.749 [2.777]
-‐6.322 [5.540]
0.438 [1.331]
Observations 716 716 337 538 444 Robust standard errors in brackets * significant at 5%; ** significant at 1%
39
-‐10
0
10
20
30
40
50
-‐2
-‐1
0
1
2
3
4
5 Four Years Before any AP Policy
Three Years Before any AP Policy
Two Years Before any AP Policy
One Year Before any AP Policy
First Year with an AP Policy
Second Year with an AP Policy
Third Year with an AP Policy
Four+ Years with an AP Policy
Increase in AP Students relative to
5 Years Before Any AP Policy
Increase in College Enrollment Rate
Relative to 5 Years Before Any AP Policy
Graph of Table 2: Effect of Any AP Policy on Number of AP Students and College Enrollment
for Seniors
Percentage of High School Graduates Enrolling Immediately in College
Number of AP Students Per 1,000 HS Juniors and Seniors
40
Table 4: Effects of Specific Policies (1994-2009) (Specification 2: Year Fixed Effects and State Fixed Effects) (1)
AP Students Per 1,000 HS Juniors and Seniors
(2) AP Exams
Per 1,000 HS Juniors and Seniors
(3) AP Exams Passed Per 1,000 HS
Juniors and Seniors
(4) Percentage of High School
Graduates Enrolling in College
Immediately After HS
Graduation
(5) Three-‐Year College
Graduation Rate of
Associate’s Degree Seeking Students
(6) Six-‐Year College
Graduation Rate of
Bachelor’s Degree Seeking Students
Equipment/ Instructional Grants
-‐26.330 [14.482]
-‐14.796 [8.186]
-‐2.831 [5.044]
1.259 [2.609]
0.109 [2.172]
0.439 [0.592]
Financial Incentives
36.604* [15.644]
22.121** [7.608]
18.882** [4.946]
3.081 [2.073]
4.804* [2.359]
0.046 [0.530]
Funds for Teacher Training
22.197 [13.854]
12.319 [6.931]
5.31 4.296]
2.824* [1.342]
-‐0.102 [1.319]
1.788** [0.572]
State Test Fee Reductions
-‐8.206 [17.638]
-‐4.706 [9.656]
-‐3.009 [4.015]
-‐2.845* [1.266]
-‐0.343 [1.865]
-‐0.705 [0.510]
Guaranteed Minimum AP Credit
-‐1.738 [15.699]
0.561 [8.351]
-‐6.911 [5.329]
-‐2.995 [1.554]
0.407 [1.718]
2.628** [0.707]
Accountability Incentives
39.593 [22.212]
21.138 [11.141]
6.292 [6.454]
0.787 [0.993]
1.438 [1.670]
0.693 [0.698]
Mandating AP Offerings
47.444 [38.053]
29.1 [23.342]
9.412 [11.417]
2.436 [1.610]
2.228 [3.417]
-‐0.144 [0.545]
K12 and Higher Ed Collaboration
11.115 [14.740]
7.101 [7.612]
-‐3.17 [6.267]
0.862 [1.584]
-‐0.562 [1.598]
0.367 [0.859]
National Test Fee Reductions
-‐7.245 [4.901]
-‐4.401 [2.914]
-‐0.397 [1.605]
-‐0.109 [0.700]
-‐0.543 [0.478]
0.113 [0.286]
National AP Incentive Grants
26.830* [12.176]
13.064* [5.710]
5.039 [5.367]
-‐1.223 [0.892]
-‐0.317 [1.590]
-‐0.142 [0.758]
Observations 716 716 385 337 538 444 Robust standard errors in brackets * significant at 5%; ** significant at 1%
41
Table 5: Effects Over Time of the Effective AP Policies (1994-2009) AP Outcomes Effective Policies: Financial Exam Performance Incentives and National AP Incentive Grants (1)
AP Students Per 1,000 HS Juniors and Seniors
(2) AP Exams Per 1,000 HS
Juniors and Seniors
(3) AP Exams Passed Per 1,000 HS
Juniors and Seniors
(4) Percentage of High School
Graduates Enrolling in College
Immediately After HS
Graduation
(5) Three-‐Year College
Graduation Rate of
Associate’s Degree Seeking Students
(6) Six-‐Year College
Graduation Rate of
Bachelor’s Degree Seeking Students
First Year of Effective Policy
3.163 [5.994]
8.054 [11.383]
5.875 [4.660]
1.472 [1.498]
0.974 [1.311]
0.839 [0.513]
Second Year of Effective Policy
2.257 [6.898]
6.808 [13.238]
-‐0.385 [5.964]
-‐2.445* [1.113]
-‐0.811 [1.675]
1.128 [0.728]
Third Year of Effective Policy
-‐4.013 [8.813]
-‐5.054 [16.339]
5.937 [6.398]
1.888 [1.195]
0.084 [1.890]
0.720 [0.945]
Four+ Years of Effective Policy
24.839* [12.103]
48.482 [24.674]
19.178* [8.370]
0.176 [1.634]
3.089 [2.157]
2.068 [1.282]
Observations 716 716 385 337 538 444 Robust standard errors in brackets *significant at 5%; ** significant at 1%
42
Table 6: Effects Over Time of the Effective AP Policies (1994-2009) College Outcomes Effective Policies: Funds for Teacher Training and Guaranteeing Minimum AP Credit (1)
AP Students Per 1,000 HS Juniors and Seniors
(2) AP Exams Per 1,000 HS
Juniors and Seniors
(3) AP Exams Passed Per 1,000 HS
Juniors and Seniors
(4) Percentage of High School
Graduates Enrolling in College
Immediately After HS
Graduation
(5) Three-‐Year College
Graduation Rate of
Associate’s Degree Seeking Students
(6) Six-‐Year College
Graduation Rate of
Bachelor’s Degree Seeking Students
First Year of Effective Policy
0.349 [6.069]
1.397 [11.376]
1.964 [3.674]
2.127 [2.098]
0.309 [1.335]
1.455** [0.486]
Second Year of Effective Policy
-‐0.698 [7.119]
-‐1.658 [13.825]
4.749 [4.583]
1.172 [1.654]
0.034 [1.541]
1.883** [0.878]
Third Year of Effective Policy
6.666 [7.820]
10.328 [14.882]
7.011 [5.753]
2.158 [1.315]
0.886 [2.252]
2.768** [0.594]
Four+ Years of Effective Policy
13.221 [8.137]
21.727 [16.181]
6.140 [7.509]
1.909 [1.696]
0.131 [2.386]
3.136** [0.896]
Observations 716 716 385 337 538 444 Robust standard errors in brackets *significant at 5%; ** significant at 1%
43
Appendix A: State Policies and Timing
Table A1: Year of First AP Policy by State State Year of First AP Policy South Carolina 1987 Missouri 1987 Utah 1988 West Virginia 1989 California 1990 Indiana 1990 Iowa 1991 Wisconsin 1993 Arkansas 1995 Montana 1995 New Jersey 1995 Massachusetts 1996 Michigan 1996 Texas 1996 Georgia 1997 Oklahoma 1997 Colorado 1998 Ohio 1998 Louisiana 1998 Virginia 1998 Kentucky 1998 South Dakota 1998 Washington 1998 Delaware 1998 Oregon 1998 Alabama 1998 Mississippi 1998 New Mexico 1998 Tennessee 1998 Hawaii 1998 New York 1998 Pennslyvania 1998 Rhode Island 1998 Florida 1999 Minnesota 1999 Connecticut 1999
44
Maryland 1999 Illinois 1999 Maine 1999 North Carolina 1999 Nevada 2000 New Hampshire 2000 Vermont 2000 Alaska 2000 Kansas 2000 Nebraska 2000 Wyoming 2000 North Dakota 2001 Arizona 2003 Idaho 2003
Table A2: Variation in Timing of State Advanced Placement Policies
Policy State and Year9 Equipment or Instructional Grants to
Schools or Districts Missouri (1986) Indiana (1990)
West Virginia (1991)10 Arkansas (1995) Oklahoma (1997)
Mandating Course Offerings South Carolina (1984)11 West Virginia (1991) Indiana (1994)12 Ohio (1998)13
Arkansas (2005)14 Mississippi (2006-‐2009)15
Tennessee (2008) Accountability Incentives
1. Must report number or percentage of
Missouri (1986): 1, 2 Wisconsin (1993): 1 New Jersey (1995): 1
9 Year in parentheses is the year the policy takes effect; unless otherwise noted, the policy continues to the present. 10 Schools in Wyoming receive extra funding for each pupil enrolled in an Advanced Placement course. 11 Each district must provide at least one AP course if there is an adequate number of talented students. 12 Each district must offer at least 2 AP courses. 13 Each educational center in Ohio must develop a plan to guarantee AP offerings. 14 All districts in the state must offer AP courses. 15 All schools must offer at least one AP course in each subject area.
45
students enrolled in AP 2. Must report number of courses offered 3. Must report the number or percentage of
students who take AP exams 4. Must report AP exam performance 5. Accreditation and school grades depend on
16 Schools receive $50 for each student who earns a 3 or higher on the AP exam. 17 Districts receive additional funding for each student who earns a 3 or higher on the AP exam. 18 The best performing schools in the state (based on the percentage of students passing exams and percentage of students who successfully complete an AP) receive financial awards.