Copyright ©2006. Battelle for Kids. Understanding & Using Value-Added Analysis
Jan 12, 2016
Copyright ©2006. Battelle for Kids.
Understanding & UsingValue-Added Analysis
Learning Targets
• Develop confidence in accessing, navigating and interpreting value-added information.
• Understand the connections between value-added information and school improvement.
Introductions
• Name
• Role in Building and/or District
• Draw a face on a post it to indicate your feelings about Value Added
The Work of the Day• Morning will be spent learning about
value-added information:– Conceptual understanding of value-added– Overview of the information available in value-
added reports
• Afternoon will be spent:– Navigating through value-added reports– Learning about people and tools that can
support for your learning– Connecting value-added to school
improvement
Copyright ©2006. Battelle for Kids.
Getting Started Value-Added Analysis
Copyright ©2006. Battelle for Kids.
Presentation Targets
• Gain an introductory understanding of value-added analysis.
• Acquire an awareness of information available in value-added reports.
• Explore how value-added information can inform school improvement.
• Understand the difference between progress and achievement.
Copyright ©2006. Battelle for Kids.
Stair-Step Expectations
Grade Three Grade Four Grade Five
Grade 3 Test Grade 4 Test Grade 5 Test
In a perfect world scores on year-end achievement tests would be enough to document growth. Students would start school at the same place, and move together toward proficiency.
Copyright © 2003. Battelle for Kids
Copyright ©2006. Battelle for Kids.
Differentiated Reality
Grade Three Grade Four Grade Five
Students begin school at different places and progress at different rates. This complicates accountability. We need more than scores on a single test to tell the story of a school’s effectiveness.
Grade 3 Test Grade 4 Test Grade 5 Test
Copyright © 2003. Battelle for Kids
Copyright ©2006. Battelle for Kids.
Achievement measures alone provide a partial picture:School improvement perspective
Birth Kindergarten 4th Grade Test
Achievement measures are snapshots in time. They provide information on what a child has learned since birth.
If we want to improve schools we need a metric that provides reliable feedback on the work that schools produce.
1st 2nd 3rd
Copyright ©2006. Battelle for Kids.
A View of Student Progress:Accountability perspective
30
40
50
60
70
80
90
100
3 4 5 6 7 8
Student AStudent BStandard
We must expect progress for all students.
Student A
Student B
Proficient
Copyright ©2006. Battelle for Kids.
The question for educators today
Regardless of where students begin, how do we maximize the progress each
makes in any given year?
84% of the American public believe student progress is
the best measure of a school’s effectiveness!
Copyright © 2003. Battelle for Kids
Copyright ©2006. Battelle for Kids.
What is the difference between progress and achievement?
Copyright ©2006. Battelle for Kids.
Value-Added Analysis in Ohio
Project SOAR
Ohio’s Value-Added
System
Copyright ©2006. Battelle for Kids.
• Operated by the Ohio Department of Education.
• Began as a grade 4 pilot in 2006 in all districts and community schools.
• Provides analysis in math & reading, grades 4-8.
• Uses only state achievement test data.
• Uses the mean gain model.
• Expected growth is set by the State Board of Education.
Ohio’s System
Copyright ©2006. Battelle for Kids.
• Operated by Battelle for Kids.
• Began in 2002 with 42 school districts.
• Now includes 100+ districts.
• Provides analysis in all core subjects, grades 3-10.
• Uses state and non-state test data.
• Uses the mean predicted model.
• Expected growth is normative (Average Growth).
Project SOAR (Schools’ Online Assessment Reports)
Copyright ©2006. Battelle for Kids.
Why EVAAS™?
• Applies the most sophisticated statistical
methodology available to ensure reliability.
• Allows for the use of all student test data.
• Provides valuable diagnostic information.
• Accommodates different types of test data.
• Used statewide in Tennessee for more than 10 years.
Copyright ©2006. Battelle for Kids.
Ohio’s Scale Up Plan
R M Sc SSR M Sc SS
Value-Added in the State Accountability System
Copyright ©2006. Battelle for Kids.
• If a district or school demonstrates more than expected growth over two consecutive years it moves up one level in the system.
• If a district or school demonstrates less than expected growth over three consecutive years it moves down one level in the system.
• Value-added measures are significant at one standard error.
Copyright ©2006. Battelle for Kids.
How is student assessment data used to produce a value-added growth metric?
Value-Added Analysis*Conceptual Overview
Copyright ©2006. Battelle for Kids.
What is a Growth Measure?
Growth = Observed Score – Baseline Score
Copyright ©2006. Battelle for Kids.
Value-Added Example
4th 5th 6th 7th 8th
A value-added metric measures the average growth that is produced in one subject in a given year.
This growth is then compared to the amount of growth that would be expected for that group of students. Growth above (or below) expectation is the “value-added.”
Copyright ©2006. Battelle for Kids.
Tests must:
• Be highly correlated to curricular objectives.
• Have sufficient stretch.
• Have appropriate levels of reliability.
Test Properties
Copyright ©2006. Battelle for Kids.
School Effect ≈ Mean Gain – Expected Growth
Used when all tests across grade levels have consistent scales.
Mean Gain Approach
Copyright ©2006. Battelle for Kids.
Mean Gain Example
A crude measure of the growth for this group is 10 NCEs.
Growth = Mean Observed – Mean BaselineGrowth = 35.2 - 25.2 = 10.0
Math scaled scores for the same students.Year 1 • Grade 3Student 1 377 = 26 (NCE)Student 2 380 = 27Student 3 360 = 19Student 4 377 = 26Student 5 381 = 28
Mean Baseline = 25.2
Year 2 • Grade 4Student 1 400 = 37 (NCE)Student 2 385 = 30Student 3 395 = 35Student 4 405 = 41Student 5 390 = 33
Mean Observed = 35.2
Copyright ©2006. Battelle for Kids.
Mean Gain Approach (Converted to NCEs)
High Progress Positive School Effects
High Achievement High Mean NCE or Mean Scaled Score
School Effect
Observed
Baseline
Sample School Value-Added Report Math
Copyright ©2006. Battelle for Kids.
Mean Gain Approach (Converted to NCEs)
Copyright ©2006. Battelle for Kids.
Describe, in conceptual terms, the “mean gain” value-added model.
What NewDiagnostic Information DoValue-AddedReportsProvide?
Copyright ©2006. Battelle for Kids.
Value-Added Summary Report
Where are the areas of strength and weakness?
Copyright ©2006. Battelle for Kids.
Are the curriculum and instruction working for all students?
School Diagnostic Report
Copyright ©2006. Battelle for Kids.
What pattern would represent optimum growth?
Performance Diagnostic Report
Copyright ©2006. Battelle for Kids.
How does a student’s growth rate compare to other groups of students?
Student Report
Copyright ©2006. Battelle for Kids.
With average growth, where will this student likely score in the future?
Student Projection
Student Search
Copyright ©2006. Battelle for Kids.
How can Predictions help identify at risk students?
Copyright ©2006. Battelle for Kids.
How could this information change the way educators approach intervention?
Student Search Results
Copyright ©2006. Battelle for Kids.
In Summary, Value-AddedInformation Shows…
• The students’ progress made in each subject area and grade level.
• The progress students made who enter the year at different achievement levels.
• How students’ progress in one curricular area or program compares to their progress in another.
• Whether individual students are making adequate progress toward state standards.
Without data, all we have are opinions!
Copyright ©2006. Battelle for Kids.
What new information do value-added reports provide?
Copyright ©2006. Battelle for Kids.
Why is it important to focus school improvement strategies at the individual grade level and/or subject area?