NEXT-GENERATION ACCOUNTABILITY Designing a Differentiated Accountability System for Michigan Presentation to the Michigan Educational Research Association.

NEXT-GENERATION ACCOUNTABILITYDesigning a Differentiated Accountability System for Michigan

Presentation to the Michigan Educational Research AssociationNovember 22, 2011

The Challenge

• Design a school accountability system that:• Sets a high proficiency standard (where proficiency is

based on career and college ready standards) AND• Rewards schools for achieving growth with students,

regardless of starting point• Moves Michigan toward a higher level of preparation for

career and college• Fair and equitably applied

WHERE ARE WE NOW?Assessing Michigan’s Current Situation

College going rates

• Statewide: • 71% of 2008-2009 graduates enrolled in an IHE• 73% of those who enroll earn at least one year’s worth

of credits• Gives a total of 52% of 2008-09 graduates who earned

at least one year’s worth of credits

• By individual school:• Median = 63%• 25th percentile: 40%• 75th percentile: 75%

0.5

11.

52

2.5

Den

sity

0 .2 .4 .6 .8 1

Percentage of Graduates Enrolled at an IHE

Relationship between new cut scores and college going0

2040

6080

100

Per

cen

t E

nro

lled

in a

n IH

E

0 20 40 60 80 100

2008 MME Math Percent Proficient (based on new cut scores)

Takeaways

• Michigan students are going to college• Even if students are not proficient on new cut scores on the MME, they are enrolling in college.

Question: Will those students be successful? Will they pursue challenging majors?

Achievement Gap

• Since 2001, schools have been held accountable on overall student performance… AND the performance of the nine traditional subgroups

• Put the focus on achievement of all students, as defined by demographic characteristics

• Caveat: IF you had a sufficient number of students!

Economically disadvantaged gaps: Math

Ethnicity gaps: Math

Economically disadvantaged gaps: Reading

Ethnicity gaps: Reading

Career and College Readiness in Our Schools: Math

0.0

1.0

2.0

3.0

4

Den

sity

0 20 40 60 80 100mmemath2011

50th percentile 95th percentile

CCR: Reading0

.01

.02

.03

Den

sity

0 20 40 60 80 100mmeread2011

50th percentile 95th percentile

Tension: Is it really important that our students be career and college ready?

• Evidence points to the importance of higher education.

• Competitive job market, especially in states like Michigan.

• Fast-growing occupations (health care, technology) require higher education/specific training

• Educational inflation

NEXT GENERATION ACCOUNTABILITYCharting a New Path

Necessary Components

• Focused consequences and interventions for schools most in need

• Achievement gap• Differentiated accountability = differentiated interventions

• Fair and equitable• Ambitious AND achievable goals

Focused Consequences and Interventions

• Priority Schools• Lowest 5% of the Top to Bottom list• Priority schools = PLA Schools• Aligns federal and state accountability• Priority schools must enter a three year cycle of school

improvement, with the most highly targeted interventions

PLA Schools: Anecdotal Evidence

• Two “cohorts” of PLA schools: 2010 and 2011.• 2010 schools: first year of implementation• 2011 schools: planning• 2010 schools:

• About half experienced an increase in percent proficient and increased their improvement rate

• More meaningful data in another 1-2 years

PLA Schools: Anecdotal Evidence

• What are they saying? • ERA Unit doing PLA data visits• Being named a PLA school was extremely difficult• However, it has fundamentally altered the way the

schools are approaching achievement• Pushing a fundamental redesign• Impetus to address crucial issues• Innovative strategies (i.e. flipped learning)

Achievement Gap as Central Focus

• Achievement gaps have not closed to the extent that we need

• Proposal:• Focus on the bottom 30% of students, regardless of

demographic, not the traditional subgroups• Puts the attention firmly on the lowest achieving

students• By improving that group, increase school’s overall

achievement, and improvement rate

Achievement Gap

• Pros• All schools have a subgroup• At least 700 schools have no subgroup under AYP

traditional subgroups• Unmask low performance in high performing subgroups• Asks that all schools consider their lowest performing

students• Schools cannot mask low-performance with overall high

performance

Achievement Gap

• Cons:• Concern that we will lose focus on demographic

subgroups.• In the lowest 30% subgroup—approximately 70% of

that group are also a member of one or more traditional subgroups.

• High-achieving schools do not like it• People think that “lowering the ceiling on our highest

achieving students” will help the schools

Achievement Gap: Focus Schools

• Need to identify the schools with the largest achievement gaps.

• Using the bottom 30% subgroup, would rank the bottom 10% in terms of largest gap.

• Using traditional subgroups in a ranking (normative) setting is complicated:• Not all schools have a subgroup• Comparing schools with the same subgroup• Unfairly focuses on students with disabilities• Still allows for “masking”

Achievement Gap: Our Belief

• If Michigan is serious about raising the achievement of ALL students, then the bottom 30% is the correct way to go.

• Distributing accountability to traditionally high achieving schools and asking them to achieve those same results with all students is appropriate.

• Michigan cannot leave students behind any longer.

Differentiated Accountability For All Schools

• Priority schools = 5% and Focus Schools = 10%--so what about the other 85%?

• Need a more nuanced system than pass/fail AYP.• Need to integrate performance for all students, bottom 30%, and all subjects (not just reading and mathematics)

Use this system to set a proficiency goal with improvement• Proficiency target = AMO

• Set for each school as the increase in percent proficient necessary for that school to reach the overall target proficiency

• Improvement target• If school does not meet proficiency target, can meet an

improvement target• Set as the increase in percent proficient demonstrated

by a high-improvement school in the base year

Example with Data

• End proficiency target: 85% • School is at 10% proficient now• Need to improve 75% in 10 years, or 7.5% per year.

• Proficiency target in year 1: 17.5% proficient• If does not meet it, must have improved by 3.5% (which is improvement rate for school at the 90th percentile in base year)

Bottom 30% Subgroup as Accountable Subgroup

• Only one “accountable” subgroup now (still report on nine traditional subgroups)

• Need to meet a proficiency target for the bottom 30% subgroup (unlikely…) OR the improvement target

Notes on this system

• Offers differentiated AMOs by school• Keeps a clear proficiency target in the system• Proficiency target is actually an improvement target as well

• Many of the increases demanded of schools will be greater than we have historically seen, so need the improvement (safe harbor) target

Questions

• What is an “ambitious and attainable” end goal? 100%? 85%? 70%? How do you determine this?

• Should meeting the target based on improvement be equivalent to meeting it based on straight proficiency?

• Should we reset each year?

Participation

• Necessary to keep a firm participation target in the system

• Schools will begin to “game” on who they assess if no clear participation target

“Green” SchoolSchool Name: ABC Schools

 Reward

   Proficienc

y Improvement Participation Overall Math All Students Yes -- Yes

   Bottom 30% No Yes  Reading All Students Yes -- Yes

   Bottom 30% Yes --  Writing All Students Yes -- Yes

   Bottom 30% No Yes  Science All Students Yes -- Yes

   Bottom 30% No Yes  Social Studies All Students Yes -- Yes

   Bottom 30% Yes --      Rate Improvement OverallGraduation Rate   Yes --              OverallCompliance          Educator Evaluations          

“Yellow” SchoolSchool Name: XYZ Schools

   Proficienc

y Improvement Participation Overall Math All Students No Yes Yes

   Bottom 30% No Yes  Reading All Students Yes -- Yes

   Bottom 30% Yes --  Writing All Students No Yes Yes

   Bottom 30% No Yes  Science All Students Yes -- Yes

   Bottom 30% No Yes  Social Studies All Students Yes -- Yes

   Bottom 30% Yes --      Rate Improvement OverallGraduation Rate   Yes --              OverallCompliance          Educator Evaluations          

“Red” SchoolSchool Name: MNO Schools

 Priority

    Proficiency Improvement Participation Overall Math All Students No Yes Yes

   Bottom 30% No No  Reading All Students No No Yes

   Bottom 30% No No  Writing All Students No Yes No

   Bottom 30% No Yes  Science All Students Yes -- Yes

   Bottom 30% No No  Social Studies All Students No No Yes

   Bottom 30% No No      Rate Improvement OverallGraduation Rate   No Yes              OverallCompliance          Educator Evaluations          

Rules for Colors

• Need to be green on all indicators• This makes “green” a more rare indicator; it means that there are no areas of concern

• To be red, need to be red on all five academic indicators; makes it a more rare indicator

• Yellow—largest category—can have some red, some green; is indicative of “intervention” needed; use colors within to target

• Final color is not the key determiner for consequences; priority/focus status is more critical

Questions?

• How to determine the final colors?• Balancing public desire for “one” rating with internal knowledge that “one” rating is difficult.

• Other indicators that should be included?

Contact Information

• Venessa A. Keesler, Ph.D.• Evaluation, Research and Accountability• Office of Psychometrics, Accountability, Research and

Evaluation (OPARE)• [email protected]