Top-to-Bottom Top-to-Bottom Ranking Ranking & & Priority/Focus/Rewa Priority/Focus/Rewa rd rd Designations Designations Understanding the
Top-to-Bottom Top-to-Bottom Ranking Ranking
&&Priority/Focus/RewarPriority/Focus/Rewar
ddDesignationsDesignations
Understanding the
Why the TTB Ranking?Why the TTB Ranking?• Move from metric as designation only
(“stick”) to leveraging the metric as a diagnostic tool for schools
• Resist urge for “more data” until we understand the metrics available; avoids “dying in data”
• Focus of this presentation:o Overview of calculationso How to interpret results (for schools and
districts)
How Designations are How Designations are RelatedRelated
The Top-to-Bottom ListThe Top-to-Bottom List• Statewide ranking of ALL schools that meet ranking
criteria
• Bottom 5% = Priority (PLA) Schools
• 10% schools with largest achievement gaps = Focus Schools
• Also used for Reward School status:o Use top 5% from overall ranking = Reward
Schoolso Use top 5% improving schools = Reward Schoolso Use Beating the Odds Schools = Reward Schools
Top to BottomTop to Bottom(TTB) Ranking(TTB) Ranking
• Three main components by subject:
• Achievement
• Improvement in achievement over time (top 5% here become Reward Schools)
• Achievement gap top 30% vs. bottom 30% of students (bottom 10% here become Focus Schools)
• Each component tells schools something about their overall performance and can be used for diagnostics
TTB RankingTTB Ranking• Graduation rates are included in the statewide
Top-to-Bottom Ranking.
• Schools with a graduation rate have it included in the following two ways:
• Graduation Rate
• Improvement in graduation rate over time
Who Receives a Who Receives a Ranking?Ranking?
• Schools with 30+ full academic year (FAY) students over the last two years in at least two state-tested content areas; school must be OPEN at time of list generation
• Application Some schools do not receive a ranking if they:
• Have too few FAY students• Only have one year of data
Tested Grades & Tested Grades & SubjectsSubjects
• Reading and Mathematics: Grades 3-8 and 11• In grades 3-8, testing every year allows us
to figure out student performance level change (our current “growth” metric) in reading and math
• Students can either significantly improve, improve, maintain, decline or significantly decline
• Writing: Grades 4 & 7• Science: Grades 5 & 8• Social Studies: Grades 6 & 9
What About What About Reconfigured Schools?Reconfigured Schools?
• A school must change by four or more grades in order to get a new code
• Example: A K-2 building becoming a K-6 building.• New codes NOT granted when a school is reopened as a charter,
for example
• If not, the school retains the old code and continues to have data “point” at it from all students for whom that code is their feeder school
• There is no “phase reset” like there was in AYP• If school population changed by 51%, could request a phase reset—
still got AYP calculations, but sanctions delayed• Under Priority/Focus interventions, would simply have a customized
intervention.
How Is the Top to Bottom How Is the Top to Bottom Ranking CalculatedRanking Calculated
• For grade 3-8 reading and mathematics
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Performance Level
Change Index
Two-Year Average Performance Level
Change Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Level Change
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
score
How do we get How do we get Standardized Scale Scores Standardized Scale Scores
for Each Student?for Each Student?• Step #1: Take each student’s score on the test
they took and compare that score to the statewide average for students who took that same test in the same grade and year
• This creates a student-level z-score for each student in each content areao Compare
• MEAP to MEAP• MEAP-Access to MEAP-Access• MME to MME• MI-Access
o Participation to Participationo Supported Independence to Supported Independenceo Functional Independence to Functional Independence
What do we do with those What do we do with those Standardized Scores?Standardized Scores?
• Step #2: Once each student has a z-score for each content area (based on the test they took), we take all of the students in a each school, and rank order the students within the school.o Z-scores will have come from different tests, and compare students to
statewide average for that grade, test, and subjecto But they can now be combined for the school
• Step #3: Add up all z-scores and take the average. This is now the average standardized student scale score.
• Step #4: Define the top and bottom 30% subgroups, based on that rank ordering.
Student Test Taken Z-score
Tommy Mi-Access, Participation 2.5
Sally MEAP 2.0
Maura MI-Access, SI 1.9
Fred MEAP 1.5
Ichabod MEAP-Access 1.0
Freud MEAP 0.8
Maybelle MI-Access, FI 0.7
Destiny MEAP 0.5
Harold MEAP -0.2
Bickford MI-Access, FI -0.5
Talledaga MEAP-Access -0.7
Francine MEAP -1.2
Joey MEAP -1.9
William MEAP -2.2
Student Test Taken Z-score
Tommy Mi-Access, Participation 2.5
Sally MEAP 2.0
Maura MI-Access, SI 1.9
Fred MEAP 1.5
Ichabod MEAP-Access 1.0
Freud MEAP 0.8
Maybelle MI-Access, FI 0.7
Destiny MEAP 0.5
Harold MEAP -0.2
Bickford MI-Access, FI -0.5
Talledaga MEAP-Access -0.7
Francine MEAP -1.2
Joey MEAP -1.9
William MEAP -2.2
Average Z-score (average standardized student scale
score): 0.28(sum all z-scores, divide by 14)
Student Test Taken Z-score
Tommy Mi-Access, Participation 2.5
Sally MEAP 2.0
Maura MI-Access, SI 1.9
Fred MEAP 1.5
Ichabod MEAP-Access 1.0
Freud MEAP 0.8
Maybelle MI-Access, FI 0.7
Destiny MEAP 0.5
Harold MEAP -0.2
Bickford MI-Access, FI -0.5
Talledaga MEAP-Access -0.7
Francine MEAP -1.2
Joey MEAP -1.9
William MEAP -2.2
Top 30%
Bottom 30%
How Is the Top to Bottom How Is the Top to Bottom Ranking Calculated?Ranking Calculated?
• For grade 3-8 reading and mathematics
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Performance Level
Change Index
Two-Year Average Performance Level
Change Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Level Change
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
score
What is Important What is Important to Show Schools?to Show Schools?
• For grade 3-8 reading and mathematics
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Performance Level
Change Index
Two-Year Average Performance Level
Change Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Level Change
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
scoreStep #1: Achievement
How well did the school do in that subject? Positive number = better than average
Near zero = averageNegative number = worse than average
What is Important What is Important to Show Schools?to Show Schools?
• For grade 3-8 reading and mathematics
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Performance Level
Change Index
Two-Year Average Performance Level
Change Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Level Change
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
score
Step #2: ImprovementIs the school improving in that subject?
Positive number = greater rate of improvement than average
Near zero = average improvementNegative = slower rate of improvement than
average; can also mean they are declining
• For grade 3-8 reading and mathematics
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Performance Level
Change Index
Two-Year Average Performance Level
Change Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Level Change
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
score
What is Important What is Important to Show Schools?to Show Schools?
Raw value is also meaningful:Positive number: More students improving
than decliningNegative number: More students declining
than improving
• For grade 3-8 reading and mathematics
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Performance Level
Change Index
Two-Year Average Performance Level
Change Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Level Change
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
score
What is Important What is Important to Show Schools?to Show Schools?
Step #3: Achievement GapIs the gap in that subject between top 30%
and bottom 30%:(positive number) = smaller gap than
average(negative number) = larger gap than
average(near zero) = average gap
Once you have Looked at Once you have Looked at each Component, Discuss:each Component, Discuss:
• What’s the overall pattern?o Low achievement? o Declining achievement?o Large gaps?
• Where are the actionable areas?o Which subjects need the most attention?o Is everyone doing poorly (small gap, low
achievement) or are some students doing well and others falling behind (decent achievement, but large gap)
• For grade 3-8 reading and mathematics
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Performance Level
Change Index
Two-Year Average Performance Level
Change Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Level Change
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
score
Focus Schools
Reward Schools (for improvement)
What is Important What is Important to Show Schools?to Show Schools?
Weighted Performance Weighted Performance Level Change (PLC)Level Change (PLC)
• A weighted composite of individual student performance level change is used to calculate improvement in grades 3-8 reading and mathematics
• Rewards large improvements more heavily, rewards maintenance of proficiency if a student was already proficient
Previous Proficiency
Significant Decline
Decline MaintainImproveme
nt
Significant Improveme
nt
Not Previously Proficient
-2 -1 0 1 2
Previously Proficient -2 -1 1 1 2
How is the Top-to Bottom How is the Top-to Bottom Ranking Calculated?Ranking Calculated?
• For science, social studies, writing, and grade 11 all tested subjects
Two-Year Average Standardized Student
Scale (Z) Score
Two-Year Average Standardized Student
Scale (Z) Score
Four-Year Achievement Trend
Slope
Four-Year Achievement Trend
Slope
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School AchievementZ-Score
School AchievementZ-Score
School Performance Achievement Trend
Z-Score
School Performance Achievement Trend
Z-Score
School Achievement Gap Z-Score
School Achievement Gap Z-Score
School Content
Area Index
School Content
Area Index
1/21/2
1/41/4
1/41/4Content Index Z-
score
Content Index Z-
score
How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?
• For graduation rate
Two-Year Average Graduation Rate
Two-Year Average Graduation Rate
Four-Year Graduation Rate Trend Slope
Four-Year Graduation Rate Trend Slope
School Graduation Rate Z-Score
School Graduation Rate Z-Score
School Graduation Rate Trend
Z-Score
School Graduation Rate Trend
Z-Score
School Graduation Rate Index
School Graduation Rate Index
2/32/3
1/31/3Grad
Index Z-score
Grad Index Z-
score
How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?
• Calculating a four-year slope (e.g., graduation rate)
• Plot the school’s graduation rate for the last four years
• Plot a linear regression line through the points
• Calculate the slope of the line (gives the school’s annual improvement rate)
How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?
• Calculating a four-year slope (e.g., graduation rate)
• Plot the school’s graduation rate for the last four years
• Plot a linear regression line through the points
• Calculate the slope of the line (gives the school’s annual improvement rate)
How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?
• Calculating a four-year slope (e.g., graduation rate)
• Plot the school’s graduation rate for the last four years
• Plot a linear regression line through the points
• Calculate the slope of the line (gives the school’s annual improvement rate)
Slope = 2.3%
How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?
• Calc an overall rank for a school with a grad rate
School Graduation Rate Std Index
School Graduation Rate Std Index
School Mathematics Std
Index
School Mathematics Std
Index
School Reading Std Index
School Reading Std Index
School Science Std Index
School Science Std Index
School Social Studies Std Index
School Social Studies Std Index
School Writing Std Index
School Writing Std Index
Overall Standardized School Index
Overall Standardized School Index
18%18%
18%18%
18%18%
18%18%
18%18%
10%10%
Overall School Percentile RankOverall School
Percentile Rank
How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?
• Calculating an overall ranking for a school without a graduation rate
School Mathematics Std
Index
School Mathematics Std
Index
School Reading Std Index
School Reading Std Index
School Science Std Index
School Science Std Index
School Social Studies Std Index
School Social Studies Std Index
School Writing Std Index
School Writing Std Index
Overall School Standardized
Index
Overall School Standardized
Index
20%20%
20%20%
20%20%
20%20%
20%20%
Overall School Percentile Rank
Look at each subject index. Help schools understand which subjects are strong/weak
for them.Positive number: better than average
Negative number: below averageNear zero: near average
How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?
• Calculating an overall ranking for a school without a graduation rate and without a writing score
School Mathematics
Index
School Mathematics
Index
School Reading Index
School Reading Index
School Science Index
School Science Index
School Social Studies IndexSchool Social Studies Index
Overall School Standardized
Index
Overall School Standardized
Index
25%25%
25%25%
25%25%
25%25%
Overall School Percentile RankOverall School
Percentile Rank
Which years of data Which years of data are in the ranking?are in the ranking?
For elementary and For elementary and middle schoolsmiddle schools
• Michigan tests in the fall.• These fall tests reflect the learning of students in
the previous school year.
Fall 2012 Testing
Fall 2011 Testing
Fall 2010 Testing
Fall 2009 Testin
g
SY 2010-2011
SY 2011-2012
SY 2012-2013
For High SchoolsFor High Schools• Michigan tests in the spring• The spring test (MME and MI-
Access) measures what students have learned from grades 9, 10 and grade 11 prior to the MME testing.
What do the 2012 What do the 2012 Rankings reflect?Rankings reflect?
• For elementary/middle schools:
oMEAP and MI-Access tests from fall 2011
• For high schools:
oMME and MI-Access tests from spring 2012
Resources AvailableResources Available• List of all schools and their ranking• Individual school look-up to see school-specific results• Overview presentation with voice over• FAQ• Business rules by which the rankings were calculated• Complete data file and validation file
You can access these resources at www.mi.gov/ttb
You can also request individual assistance by calling the Evaluation, Research and Accountability unit at 517-373-1342, or emailing [email protected]
Priority School StatusPriority School Status• Schools in the bottom 5% of the Top-to-Bottom
Ranking
• MDE ensures that the number of schools identified as Priority Schools includes >=5% of the state’s Title I schools.
• MDE will also add schools with a grad rate of <60% for 3 consecutive years, and any Tier I or Tier II school using SIG funds to implement a turnaround model.
How does a school EXIT How does a school EXIT Priority School status?Priority School status?
• For a school to exit priority school status, it has to receive a Green, Lime, Yellow or Orange on the Accountability Scorecard at the close of its third year in the Priority School intervention system. • A school must either meet aggressive proficiency targets (toward 85% of
students proficient by 2022), or must have demonstrated significant improvement.
• The proficiency and/or improvement gains must be demonstrated all nine traditional ESEA subgroups as well as in the new “bottom 30%” subgroup.
How does a school EXIT How does a school EXIT Priority School status?Priority School status?
• This means that a priority school who achieves a Green, Lime, Yellow or Orange on the Accountability Scorecard and exits Priority School status has:
o Met all interim measurements of progress for priority schools (approved plan, leading and lagging indicators).
o Met proficiency and/or improvement targets on average as a school.
o Increased the proficiency rate of all traditional subgroups.
o Increased the proficiency rate of their very lowest performing students.
Focus School StatusFocus School Status• 10% of Schools with the largest achievement
gaps in scale score between the top 30% of students and bottom 30% of students within a school from the Top-to-Bottom Ranking.
• Focus schools may have high average performance overall, but have a significantly large gap, suggesting struggles addressing low achieving students
The Purpose ofThe Purpose of
Identifying Focus Identifying Focus
SchoolsSchools• Critical component to closing achievement gaps
within schools and statewide.
• Highlight where changes in teaching and learning practices need to be undertaken to respond to the learning needs of low-achieving students.
• These changes are difficult; both accountability and support need to be differentiated.
Focus Schools: Focus Schools:
Achievement GapsAchievement Gaps• Top-to-Bottom list includes a component that
examines the gap in achievement scores between top 30% and bottom 30% of students within a school
• Gaps are standardized between all students using a common assessment within a school, and then averaged for the school
• Gaps are calculated for all subject areas and combined to form a composite gap
How are Focus Schools How are Focus Schools
Identified?Identified?
• Average gap is then standardized and ranked among all schools
• The 10% of schools with the largest achievement gaps are identified as Focus Schools
Focus Schools: Common Concerns Focus Schools: Common Concerns • Are Focus Schools only high-achieving
schools?
• Are Focus Schools only high socioeconomic status schools?
• Is the bottom 30% subgroup in Focus Schools actually high performing?
• Are schools more likely to be Focus Schools if they have [fill in the blank group] kids?
Focus Schools are NOTFocus Schools are NOT
Just High Achieving...Just High Achieving...
Focus Schools are NOT ExclusivelyFocus Schools are NOT Exclusively
High (or Low) Socioeconomic High (or Low) Socioeconomic
Status...Status...0
20
40
60
80
100
Pe
rcen
t E
con
om
ically
Dis
ad
van
tage
d
0 20 40 60 80 100Overall Percentile Rank
Focus Schools Non-Focus Schools
Bottom 30% Students are Bottom 30% Students are
NOT High Achieving ...NOT High Achieving ...
0.5
11
.5
-2 -1 0 1 -2 -1 0 1
Non-Focus Focus
Den
sity
Bottom 30% Reading E/MSGraphs by Focus
0.5
11
.5
0 1 2 3 0 1 2 3
Non-Focus Focus
Den
sity
Top 30% Subgroup Reading E/MSGraphs by FocusBottom 30% Top 30%Across all subject areas and grade levels, the bottom 30% subgroup consistently had average achievement z-score below zero, and most of them are between -0.5 and -1.5. (This example: E/MS Reading)
Non-Focus SchoolsFocus Schools
Bottom 30 are Not Dominated Bottom 30 are Not Dominated
by any 1 Subgroup…by any 1 Subgroup…
Focus schools have higher concentrations of subgroups in bottom 30% than other schools
0.2
.4.6
.8Non-Focus School Focus School
Economic Disadvantage Students with DisabilitiesLimited English Proficient WhiteBlack/African American AsianHispanic Multiracial
Graphs by focusv1
Focus Designation vs. AYPFocus Designation vs. AYP• Achievement gap between
top 30% and bottom 30% of students within a school.
• This approach targets ACHIEVEMENT gaps and THEN asks the demographic question.
• Methodology detects differences in achievement within subgroups; between subgroups; or with small populations.
• Limited by the size of groups and demographic status only.
• Methodology detects differences in achievement within a subgroup as a whole or as an overall student population
Reward School StatusReward School Status
Identification as a Reward School results from achieving one or more of the following distinctions:
Being in the top 5% of the Top-to-Bottom Ranking
Being in the top 5% of improving schools from the improvement metric in the Top-to-Bottom ranking
Being a school identified as “Beating the Odds” (BTO).
Identifying Schools Beating the Identifying Schools Beating the
OddsOdds• 2 separate studies of schools Beating the Odds using
considerably different methodologies
1. Schools performing above their predicted levels based on these factors:
• Percent economically disadvantaged
• Percent students with disabilities
• Percent English language learners
• Percent minority
2. Schools performing above a comparison group of the most demographically similar schools in the state
• Provides a strong basis for concluding that these schools are indeed beating odds
What happens once a school What happens once a school
is named a Reward School?is named a Reward School?• Receive public recognition for their achievements
through a communication to local media.
• Have their practices highlighted at conferences and other events, such as MDE’s School Improvement Conference.
• MDE is seeking other supports for Reward Schools, including increased flexibility in the use of federal grant funds, corporate and philanthropic support, and networking meetings for school leaders and educators.
Summary of Summary of Ranking-Related Ranking-Related
DesignationsDesignations
We’re here to help!We’re here to [email protected]
-OR- 877-560-8378, Option 6