Independent Study of State End-of-Course Assessments Presentation of Findings House Education Committee February 1, 2008.

Independent Study of

State End-of-Course Assessments

Presentation of Findings

House Education Committee

February 1, 2008

Slide 2

Introduction

• The Washington State Board of Education (SBE) contracted with Education First Consulting to conduct an independent study of state end-of-course assessments.

• Our research team includes:– Jennifer Vranek, President of Education First Consulting– Jessica de Barros, Education Policy Consultant– Richard Brown, Psychometrician and Assistant Professor in the

USC Rossier School of Education– Betheny Gross, Senior Research Analyst in the University of

Washington’s Center on Reinventing Public Education– Christopher Mazzeo, Education Policy Consultant, Consortium

for Chicago School Research

Slide 3

Our Approach

Our team researched the following:• What lessons can Washington State learn from the

literature on end of course tests (EOCs), comprehensive assessments and exit exams?

• What have been the experiences of other states implementing EOCs?

• What alternatives measure Washington’s standards?• What are the policy implications for Washington’s

high school assessment system based on lessons learned from other states with EOCs?

Slide 4

Methodology

• First, the team conducted a thorough review of the primary and secondary literature on EOCs and comprehensive tests, as well as the use of these formats for exit exams and school accountability.

• Next, we researched and summarized nine states’ EOC programs.– CA, IN, MD, NJ, NY, SC, TN, TX and VA

• We simultaneously conducted 30 interviews of education, government and business leaders in seven states.– CA, IN, NJ, SC, TN, TX and VA

• Finally, we used our research findings to consider policy implications for Washington State’s high school assessment system.

Slide 5

What are EOCs? What is the WASL?

End-of-course assessments are tests that:– Are given at the end of a particular course and measure the

content found only in the standards for that course.

– Are taken only by students taking a particular course, so not all students are assessed.

The 10th grade Washington Assessment of Student Learning in reading, writing, mathematics and science is a comprehensive assessment because:– It is given at a common point in time (to all 10th graders) and

– It measures content and skills from standards that are taught over several years, rather than in a particular course.

Slide 6

Purposes Of Assessment

There are many ways to describe the purposes of statewide high school assessment. We summarize these as four main purposes:

– Supporting student learning;

– Holding students and/or schools accountable;

– Determining readiness for postsecondary education and training; and,

– Ensuring high-quality and efficient operations.

Slide 7

Assessments:Means to an End

• On balance, both comprehensive and EOC assessments can meet these four purposes equally well.

• However, each approach has different strengths and meets the purposes in different ways.

• The advantages and disadvantages of either test format depend on the state’s purposes for assessment.

Slide 8

Comparing Use Of Assessments

• Comprehensive assessments are more common today than EOCs…

• But more states are shifting to EOCs.– 16 states currently use EOCs– 11 additional states are planning to use EOCs

Slide 9

States with EOCs (In Place or Planned)

16 Have EOCs in place (or field-testing) in 2007-2008 school year: AR, CA, GA, IN, LA, MD, MA, MS, NJ,

NY, NC, OK, SC, TN, UT, VA

11 Report plans to have EOCs in at least one subject area: AZ, FL, HI, KY, MI, NM, OH, PA, RI, TX, WV

7 Planning to keep both EOCs and comprehensive assessments: AR, CA, GA, MA, MI, NJ, SC

Slide 10

States with EOCs as Exit Exams or for School Accountability

(In Place or Planned)

26All states currently or planning to have exit exams for students:

AL, AK, AR, AZ, CA, FL, GA, ID, IN, LA, MD, MA, MN, MS,

NV, NJ, NM, NY, NC, OH, OK, SC, TN, TX, VA, WA

10

States currently or planning to use EOCs to hold students accountable:AR (2010), IN (2000), MD (2009), MS (2006), NY (2000),

NC (2010), OK (2012), TN (2005), TX (2012), VA (2004)

3

States that will use comprehensives in English/math and EOCs in other subjects:

MA (English/math 2003, science EOCs 2010, U.S. History EOC 2012)NJ (English/math 2003, Biology EOC 2010)

SC (English/math 2006, Biology and U.S. History EOCs 2010)

12 Use or plan to use some or all of their EOCs for school accountability under NCLB:

AR, MA, MD, MS, NJ, NY, NC, OK, SC, TN, UT, VA

Slide 11

Key Findings:Comprehensive Assessments

Overall, we found that comprehensive testing systems:• Usually focus on 10th grade or lower standards;• Assess a slice of high school standards, rather than deep

knowledge of particular subjects;• Can potentially narrow the delivered curriculum to what is

tested;• Provide a “snapshot” of system performance; • Often take up less testing time overall and cost less;• Take a more straightforward approach to exit exams and

school accountability; and • Rarely provide information on students’ readiness for

postsecondary education coursework and training.

Slide 12

Key Findings:EOC Assessments

Overall, we found that EOC testing systems:• Vary widely with respect to number and kinds of courses assessed;

• Will measure a broader and deeper range of standards, but only if there are a sufficient number of EOCs in each subject;

• Do not assess all students against common standards, unless states require all students to take certain courses and/or require all students to take certain EOCs;

• Can promote more consistency of teaching and provide more timely information on learning and course quality;

• Motivate students to learn through exit exams as well as other forms of lesser student stakes (e.g. counting results in course grades;

• Make it more complicated to hold schools accountable, yet offer the potential to produce more validity and reliability; and

• Can be better-suited for placing students in postsecondary education courses than comprehensive tests given by states in the 10th grade.

Slide 13

Policy Implications for Washington

• Our research revealed that both comprehensive and EOC assessments have strengths and limitations, depending on the state’s purpose of assessment.

• We believe Washington policymakers must first determine what their policy priorities are for student assessment in order to choose the most appropriate testing format.

Slide 14

A Decision Framework For Policy Priorities

• To help policymakers compare how each test format meets various policy priorities, we developed a “decision framework” analyzing each type of assessment according to the four purposes of assessment:– Supporting Student Learning;– Holding Students and Schools Accountable;– Determining readiness for postsecondary education and

training; and– Ensuring High-Quality and Efficient Operations.

• Within each issue area, we analyzed the relative advantages and disadvantages of comprehensive assessments and EOCs according to multiple criteria.

Slide 15

Supporting Student Learning

Issue Area Advantage to…

Supporting Student Learning Measuring the breadth and depth of standards EOCs (slight) Assessing students near the point of curriculum delivery

EOCs (strong)

Assessing students with the same test Comprehensive Choice and quality of test question types No clear

advantage

Slide 16

Holding Students & Schools Accountable

Issue Area Advantage to…

Holding Students and Schools Accountable Validity and reliability of assessments EOCs (slight) Holding students accountable No clear

advantage Reporting results at the classroom or course level

EOCs

Holding schools accountable No clear advantage

Slide 17

Determining Readinessfor Postsecondary Education

Issue Area Advantage to…

Determining Readiness for Postsecondary Education & Training Measuring readiness for postsecondary education

EOCs (strong)

Providing access to rigorous courses while preserving flexibility

EOCs (slight)

Slide 18

Ensuring Quality & Efficient Operations

Issue Area Advantage to…

Ensuring Quality and Efficient Operations

Testing window and turnaround time for results

No clear advantage

Costs and time spent on testing Comprehensive Impact of administration on schools No clear

advantage Test security No clear

advantage

Slide 19

Sequencing Policy Decisions

• Assessments are a means to the end of understanding how well students, schools and districts are meeting state standards.

• It is important to consider the assessment policy within the larger context of the state’s K-12 education system and efforts to improve learning and teaching.

• Over the next several months, the statewide system of standards, curriculum, instruction, assessment and accountability is being reshaped.

• State policymakers should first address key questions about high school education policy and then determine which format for the high school assessments is most relevant.

Slide 20

RecommendedPolicy Questions

• What skills and knowledge do students need to be successful after exiting Washington’s K-12 public education system? How important is measuring readiness for postsecondary education?

• Is it important for the state to assess academic standards close to the time when students learn content or at a common point in time?

• When the new graduation requirements are adopted, what course credits will all students be expected to earn in English, math, science and social studies?

• Should the main high school assessments also be used to measure readiness for postsecondary education?

• Does the state want additional ways to hold students accountable, such as tying test results into course grades?

• Are more statewide measures needed for schools or are the four current measures sufficient for the state accountability system?

Slide 21

EOC-SpecificRecommendations

If, after addressing these questions, Washington policymakers consider transitioning to an EOC-based system, we recommend that policymakers:

– Minimize cost and development time by working in collaboration with other states to implement standards-based, criterion-referenced assessments;

– Require all students to earn a common set of course credits, and require all students to take the corresponding EOCs in these subjects, to ensure equity.

– Maintain the 10th grade WASL in reading and writing, rather than creating EOCs in these subjects.

Slide 22

Discussion

Slide 23

Back-Up Slides

Slide 24

Detailed Findings

1. Well-designed comprehensive test systems and EOC systems share many characteristics of high-quality testing systems, such as

– Measuring state standards with reliability and validity.– Assessing standards with a variety of formats (multiple-

choice, constructed-response).– Turning around test results in a timely manner.– Scoring costs may increase with EOCs, but this depends

on number of tests/test forms, types of item formats and scoring turnaround needs.

Slide 25

Detailed Findings

2. State EOC systems vary widely on the number and kinds of courses that are assessed.

– EOC systems can assess a broader and deeper range of content and skills—if there are multiple assessments in a subject area, assessments for advanced-level courses and/or multiple subject areas.

– Comprehensive tests typically focus on 10th grade standards and assess all students at a common point in time.

– Most EOC systems still measure English with a comprehensive test. – Unless states require all students to pass a certain series of courses

and/or require all students to take certain EOCs—different groups of students will be tested on different content.

– E.g.: 700,000 California students took the Algebra I EOC (a course required for graduation) in 2006, but only 300,000 took the Geometry EOC.

Slide 26

Detailed Findings

3. While both types measure standards, EOCs are typically chosen by states to promote more consistency of teaching.

– EOC exams are designed to measure the impact of a specific course on student learning.

– Comprehensive exams are designed to measure the impact of instruction over several years on student learning.

– CA, IN, TN and VA education leaders told us they administer EOCs because the standards, content, instruction, curriculum and assessment are more closely aligned than in comprehensive systems.

Slide 27

Detailed Findings

4. Changing test formats will not necessarily improve student learning of state standards or increase student performance.

– Performance on EOCs and comprehensive tests varies widely. • Student results on the high-stakes California and South Carolina 10th grade

exit exams are much higher than on those states’ EOCs.

– It not possible to conclude that test format type has any influence on the number or proportion of students meeting standards. Factors that impact test scores include:

• Content measured;• Rigor and complexity of test questions;• Performance standards (“cut scores”);• Alignment of standards and local curriculum with the assessment; and• Students’ opportunity to learn the material tested.

– Most exit exam states—regardless of test format—have high levels of student performance overall (and many have closed performance gaps), especially when results from retakes are added.

Slide 28

Detailed Findings

5. Both EOCs and comprehensive assessments can be used as exit exams. In addition, EOCs can be used as part of student course grades.

– A handful of states—including CA, IN, LA and UT—currently attach no stakes for students to their EOCs.

– Several states—including GA, NC, SC and soon TN and TX—require districts to count EOC results as a portion of final course grades (usually 15-25%).

– 13 states currently or will require students to pass at least one EOC in order to graduate. 10 of these states use only EOCs, while MA, NJ and SC use comprehensive and EOCs. This includes IN and TX, which plan to transition away entirely from their comprehensive tests.

Slide 29

Detailed Findings

6. It can be more complicated to hold schools accountable with EOC tests.

– It is more straightforward to use comprehensive tests for school accountability.

– EOCs can paint a more complex—and more reliable—picture of high school performance. More EOC tests will yield more data and have the potential for greater validity and reliability.

7. States can use EOCs to meet No Child Left Behind requirements in reading, math and science.

– 10 states currently use EOCs to determine AYP.

– Two additional states are planning to use EOCs for AYP.

Slide 30

Detailed Findings

8. EOCs are better-suited to determine student readiness for postsecondary education and training than comprehensive tests given in 10th grade.– Most states planning to use high school tests for readiness and

placement purposes are moving towards EOCs, in large part because EOCs given in advanced subjects can be used more easily to assess readiness than comprehensive tests given in the 10th grade.

– E.g. CA, LA, NJ, NY, TX and TN do or plan to use EOCs to signal readiness for postsecondary education.

– States also are adapting the SAT, ACT, Compass/Accuplacer to administer to all 11th-graders.