Sharon Ford Schattgen, Ph.D. November 7, 2012 Presentation Prepared for the Missouri Charter School Association’s Seminar for Governing Board Members A.

An Overview of the Types of Performance Data Used to Describe

Missouri Schools

Sharon Ford Schattgen, Ph.D.November 7, 2012

Presentation Prepared for the Missouri Charter School Association’s Seminar for Governing Board Members

A Close Look at the Indices that Governing Board Members Should Use to Evaluate Charter School Quality

Types of Performance Data Used to Evaluate School Quality

Missouri Assessment Program Data MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates TerraNova Median National Percentile Ranks MAP Scale Scores Averages, Arrayed by Cohort MAP Index Scores MAP Growth Estimates

Other Summative Standardized Assessment Data Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.) College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement

Scores) Local Interim Assessment Data (e.g., NWEA Assessment Data) Additional Performance Measures

Attendance, Dropout, & Graduation Rates Post-Secondary Success Data

Accountability Data Missouri School Improvement Program Annual Performance Report (MSIP APR) Information Associated with Special Designations 2

School X Demographic DataEnrollment/School Year

2007-08 2008-09 2009-10 2010-11 2011-12

Total 285 350 434 504 575

Black 270(94.7%)

332(94.9%)

407(93.8%)

472(93.7%)

540(93.9%)

Hispanic 2(.7%)

2(.6%)

2(.5%)

3(.6%)

Not reported

White 13(4.6%)

16(4.6%)

25(5.8%)

28(5.6%)

31(5.4%)

Free/Reduced Price Lunch

227(85%)

321(96.1%)

393(92%)

467(92.3%)

514(89.4%)

Free/Reduced Price Lunch Percentages for Corresponding Years:Comparison District = 72.3%, 68.7%, 83.8%, 85.7%, 87.4%; MO = 42.1%, 43.7%, 46.9%, 47.8%,

49.5%

3

Note 1: Disaggregation of data by “subgroup” is required!

4

One of the goals of the No Child Left Behind Act is to eliminate discrepancies in achievement across “subgroups” of students, which means Governing Board members should examine data that are disaggregated to allow you to see any “gaps” in performance. In other words, you need to review data that are disaggregated by racial or ethnic group, free/reduced-price lunch status, special-education status, and limited English proficiency status, and, when appropriate, gender.

Note 2: Most of the data Governing Board members will receive will not be

presented in its original format. Rather, it will likely be summarized and reformatted for the purposes of

creating an annual report.

School personnel access their data from the DESE MCDS Portal, and test publishers also provide

electronic and print reports. School Report Cards are posted at

www.dese.mo.gov5

Note 3: We assess achievement for specific purposes; here are 5 such purposes.

1. Provide summary information about what students know and can do relative to course objectives or content/process standards

2. Yield information on an interim basis to inform policymaker and/or educator decisions at classroom, school, or district level

3. Collect student-centered information to guide minute-by-minute and/or day-to-day instruction

4. Place students in specific programs/services5. Identify students who need further evaluation

Purposes 3, 4, and 5 (listed in green, italic type) yield data that are not appropriate for inclusion in a report to a Governing Board. Data collected for these purposes have utility only for educators. 6

Frequency of administration

Sco

pe

(cu

rric

ulu

m f

ocu

s) &

du

rati

on

of

cycl

e (t

ime

fram

e)

Summative

Interim (instructional, evaluative, predictive) aka “common” or “benchmark”

Formative (minute by minute and/or day-to-day integrated into lesson) aka “classroom”

Tiers of AchievementAssessmentPerie, Marion, & Gong, 2009

Incr

ea

sin

g

A Reminder: Types of Performance Data Used to Evaluate School Quality

Missouri Assessment Program Data MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates TerraNova Median National Percentile Ranks MAP Scale Scores Averages, Arrayed by Cohort MAP Index Scores MAP Growth Estimates

Other Summative Standardized Assessment Data Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.) College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement

Scores) Local Interim Assessment Data (e.g., NWEA Assessment Data) Additional Performance Measures

Attendance, Dropout, & Graduation Rates Post-Secondary Success Data

Accountability Data Missouri School Improvement Program Annual Performance Report (MSIP APR) Information Associated with Special Designations

8

ASSESSMENT DATA

Missouri Assessment Program (MAP) Grade-Level & End-of-Course (EOC) Assessment Proficiency Rates

“MAP Proficiency Rate” is the percentage of students scoring in the top two MAP achievement levels—the Proficient Level and the Advanced Level.

Following a brief introduction to the MAP, there is a sample line graph showing trends in MAP Grade-Level Proficiency Rates, a sample bar graph showing disaggregated Grade-Level Proficiency Rates for one year, and a sample table showing trends in End-of-Course Proficiency Rates.

9

Missouri Assessment Program (MAP)

• MAP includes required consecutive grade-level (3-8) assessments in Communication Arts and Mathematics and grade-span (5 and 8) assessments in Science.

• MAP also includes end-of-course (EOC) exams, including English II, Algebra I, and Biology I. See DESE website for testing requirements.

• MAP results are analyzed and reported in a variety of ways in order to inform policy and practice and to satisfy federal and state accountability requirements.

• Performance on each MAP assessment is reported in terms of four achievement levels: Below Basic, Basic, Proficient, and Advanced.

• The TerraNova tests, which yield National Percentile Ranks, are components of the MAP grade-level assessments.

• MAP results are also reported using Scale Scores (calculated by test publisher). • DESE also calculates MAP Index Scores to use in the MO School Improvement Program

(state accountability system).

10

MAP Mathematics Proficiency RatesGrade 7

11

2008 2009 2010 2011 20120

10

20

30

40

50

60

70

32.1

27.829.3

34.6

37.9

16.514.5

27.7

24.3 25.1

49.852.5

55.2 56.4

60.1

School X Comparison Missouri

12

2009 Proficiency Rates for Black Students and White Students: MAP Communication Arts Grade-Level and End-of-Course Assessments

Columbia Public Schools and Missouri

16.4

21.725.2

19.9

25.021.9

53.450.1

57.660.0 60.6

66.264.7

85.6

21.8

28.0 28.026.3

28.626.9

51.2

46.1

51.954.9

53.456.5 56.0

77.2

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Eng II EOC

Pe

rcen

t P

rofi

cie

nt/

Ad

van

ce

d

District Black District White State Black State White

MAP End-of-Course (EOC) ExamProficiency Rates

Algebra I

2009 2010 2011 2012

School X 22.2%N=72

32.8%N=49

62.3%N=50

59.9%N=50

Comparison School

22.7% 26.3% 41.7% 44.8%

Missouri 52.6% 57.3% 59.7% 56.7%

13

ASSESSMENT DATA

TerraNova Median National Percentile Ranks

TerraNova Tests are embedded within the MAP Grade-Level Assessments. Each TerraNova yields a National Percentile Rank, which allows us to compare the performance of our students to that of their national grade-level peers. To report group performance, we calculate the Median National Percentile Rank.

TerraNova Median National Percentile Ranks for multiple years are shown on the following slide.

14

TerraNova Median National Percentile Ranks

School X Communication Arts

2008 2009 2010 2011 2012

Grade 6 41 36 33 33 43Grade 7 43 39 44 43 41Grade 8 34 36 48 44 37

School X Mathematics

2008 2009 2010 2011 2012

Grade 6 28 21 27 22 32Grade 7 29 33 29 31 31Grade 8 49 45 56 29 36

School X Science (Grade 8)2008: 39

2009: 302010: 372011: 362012: 37

NATIONAL Terra Nova Median Percentile Rank = 50; Range = 1 to 9915

ASSESSMENT DATA

MAP Scale Score Averages, Arrayed by Cohort

When arrayed by cohort, MAP Scale Score Averages show the change over time for a given group of students as they move through the grade span (e.g., sixth grade to seventh grade; seventh grade to eighth grade).

This type of analysis, shown in the following table, indicates, in a very general way, whether there is student growth in achievement.

16

MAP Scale Score Averages for Grade CohortsSchool X

Communication Arts

Grade 6 Grade 7 Grade 8

2008 650 651 674

2009 644 653 675

2010 652 655 680

2011 646 654 676

2012 653 656 671

MAP Communication Arts Scale Scores range from 455 to 875 (grades 3-8).

17

ASSESSMENT DATA

MAP Index Scores

Each year, DESE calculates a MAP Index Score for each content area in each grade span. The Index is calculated by multiplying the percentage of reportable students scoring within a given achievement level by a constant: 6 x % in Below Basic; 7 x % in Basic; 8 x % in Proficient; and 9 x % in Advanced. The resultant products are then summed to produce the Index, which ranges from 600-900. MAP Index Scores, arrayed over time, are used in the Missouri School Improvement Program as an indicator of whether there is longitudinal improvement in performance.

The following table presents an example of MAP Index Scores over time.

18

School YMAP Index Scores

Grade Level

2008 2009 2010 2011 2012 Status

Grades 3-5 Math

711.5 725.0 735.0 739.4 742.6 730.7

19

Source: Understanding Your Annual Performance Report, 2012-13. Missouri Department of Elementary & Secondary Education

ASSESSMENT DATA

MAP Growth Estimates

MAP Growth Estimates are calculated using complex statistical formulae, which utilize each student’s MAP Scale Scores over time—for example, from sixth grade to seventh grade, and from seventh grade to eighth grade. The Department of Elementary and Secondary Education first reported MAP growth data in 2011

The next slide presents an example of MAP growth data.

20

MAP Value-Added Growth EstimatesMathematics

2011School X Between

0.095 and 0.16

Significant

2012School X Between

0.082 and 0.25

Significant

State Range: -3 to +3State Average = 0

*Growth estimates were calculated using longitudinal MAP grade-level assessment data.

21

ASSESSMENT DATA

Other Summative Standardized Assessment DataStudents may take other summative standardized assessments, in addition to the MAP—assessments such as the Stanford Achievement Test or the Iowa Test of Basic Skills or ACT’s EXPLORE or PLAN. If so, these tests will likely be administered in the fall or the spring of the year. Results are used to inform instructional decisions.

A subsequent slide presents an example of data from PLAN, a standardized test that may be administered to 10th-grade students.

22

Data from Other Standardized Achievement Tests

• In addition to the MAP, students may take another summative assessment, such as Stanford Achievement Test, Iowa Test of Basic Skills, EXPLORE, or PLAN.

• Scores from these tests could be reported in terms of: – Median Percentile Rank– Percent of Students Scoring At or Above Grade– Scale Score Average

23

24

PLAN Scale Score AveragesColumbia Public Schools and Nation

2005-06 through 2008-09

1

6

11

16

21

26

31

Sca

le S

core

Ave

rag

e

District 2005-06 18.6 18.7 18.2 19.7 18.9

Nation 2005-06 16.1 16.3 15.8 17.4 16.5

District 2006-07 18.6 19.2 18.5 19.7 19.1

District 2007-08 17.8 19.0 18.0 19.9 18.8

District 2008-09 18.2 19.2 18.0 19.1 18.8

Nation 2007-09 16.9 17.4 16.9 18.2 17.5

English Mathematics Reading Science Composite

Scale: 1-32

In 2008-09, 67% of CPS studentsearned Composite Scores at or abovethe national average.

ASSESSMENT DATA

College Readiness/Placement Data

The ACT is a set of curriculum-referenced tests that measure students’ readiness for college. ACT Scale Scores range from 1 to 36, and the 2012 national average is 21.1. A student whose Scale Score in a particular content area meets or exceeds the corresponding ACT College Readiness Benchmark has a high probability of being successful in an entry-level, credit-bearing college course in that subject.

Scores from Advanced Placement Exams are used to award college credit or advanced placement in college coursework. Advanced Placement Exam cores range from 1 to 5.

The following tables present example ACT data and Advanced Placement data.

25

ACT Scale Score AveragesClass of 2012

English Math Reading Science Composite

School X 14 15.9 16.3 16 15.8

Comparison District

Data not available 16.5

Missouri 21.4 21.1 21.9 21.5 21.6

National 20.5 21.1 21.3 20.9 21.1

26

ACT Data: Class of 2012Percentage of Test-Takers

Scoring at or Above National Average Composite Scale

Score of 21

School X 20.0%

ComparisonDistrict

18.2%

Percentage of Test-Takers Scoring at or Above College Readiness

Benchmark

School X

Missouri

English: 18%Math: 13%Reading: 15%Science: 12% All 4: 12%

English: 73%Math: 46%Reading: 56%Science: 33% All 4: 27%

Percentage of 2012 graduates taking ACT: School X = 85%Comparison District = 70% MO = 67-75%US = 52%

27

28

Average Scores on Selected Advanced Placement Examinations, 2009(Exams Most Frequently Taken by District's Students)

3.48

4.38

3.21

4.07

3.93

3.74

3.74

4.02

2.92

3.54

2.76

3.31

2.92

2.88

3.14

2.67

2.88

3.22

2.72

2.99

2.64

2.84

2.83

2.71

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Lang &Comp

Psychology US History Calculus AB WorldHistory

Lit & Comp Statistics Biology

CPS n=189 CPS n=181 CPS n=143 CPS n=133 CPS n=120 CPS n=96 CSP n=74 CPS n=54

Av

era

ge

Sc

ore

District State Nation & Beyond

ASSESSMENT DATA

NWEA Assessment Indices

Some charter schools periodically administer classroom-based assessments (such as those published by NWEA, the Northwest Evaluation Association) in order to monitor student progress and to adjust instruction based on student needs.

Pre- and post-test NWEA assessment indices for the 2011-12 school year are presented on the following table; however, these data must be interpreted with care because we have no evidence of their technical characteristics (e.g., reliability and validity). However, such data could be appropriate for inclusion in a report to a board because they have the potential to provide information about student learning.

29

Example of Local Interim Assessment: NWEA Indices

Reading Math

Grade 6

Percent At/Above Grade Level, Fall 2011 42% 18%

Percent At/Above Grade Level, Spring 2012 44% 46%

Grade 7

Percent At/Above Grade Level, Fall 2011 15% 20%

Percent At/Above Grade Level, Spring 2012 40% 54%

Grade 8

Percent At/Above Grade Level, Fall 2011 16% 25%

Percent At/Above Grade Level, Spring 2012 53% 42%

Grade 9

Percent At/Above Grade Level, Fall 2011 25% 36%

Percent At/Above Grade Level, Spring 2012 66% 54%30

ADDITIONAL PERFORMANCE MEASURES

Attendance, Dropout, & Graduation Rates; Post-Secondary Success Data

Attendance rates, dropout rates, and graduation rates, along with data about post-secondary success, are shown on the following four slides. These measures of performance complement achievement data.

31

Attendance Rate2008 2009 2010 2011 2012

Example School

91.2% 92.6% 93.4% 94.7% 93.9%

Comparison District

88.6% 91.5% 92.0% 92.5% 93.1%

Missouri 94.0% 94.4% 94.2% 94.4% 94.7%

32

Dropout Rate

2008 2009 2010 2011 2012

Example School

0% 1.5% 1.6% 3.4% 7.8%

Comparison District

13.9% 13.8% 17.5% 23.6% 19.5%

Missouri 3.5% 3.5% 3.2% 3.4% 3.4%

33

Graduation Rate

Academic Year: 2011-2012

Example School 88.7%

Comparison District 74.5%

Missouri 87.7%

Graduation Rate

34

Provide trend data when possible.

Graduation RatePost-Secondary Placement Rate

35

• Over 85% of School Y’s 2010 graduates enrolled in post-secondary institutions, entered the job market in a field related to their career-education coursework, or joined the military.

ACCOUNTABILITY DATA

Missouri School Improvement Program: Annual Performance Report

The MSIP APR presents evaluative data about districts and charter schools in relation to 14 accountability standards. Schools are held accountable for only those standards pertaining to its grade span. In accordance with established criteria, the Missouri Department of Elementary & Secondary Education can issue designations based on a school’s or a district’s performance.

The following two tables summarize School X’s and a comparison district’s 2012 MSIP APRs and provide data supporting a specific designation—in this case as a “focus school.”

36

2012 MSIP Annual Performance ReportMSIP Standard/Indicator School X Comparison District

9.1.3. MAP Grades 6-8 Math Not Met Not Met

9.1.4 MAP Grades 6-8 Comm Arts Not Met Not Met

9.1.5 EOC Algebra I Math Not Met Met

9.1.6 EOC English II Comm Arts Not Met Not Met

[Bonus MAP Achievement] Not Met Not Met

[MAP Grade 8 Science] Not Met Not Met

9.3 ACT Not Met Not Met

9.4.1 Advanced Courses Not Met Met

9.4.2 Career Education Courses Not Met Met

9.4.3 College Placement Not Met Met

9.4.4 Career Education Placement Not Met Met

9.5 Graduation Rate Met Met

9.6 Attendance Rate Met Met

9.7 Subgroup Achievement Not Met Not Met37

School X “Focus School” Indices

Percentage of Students Scoring Proficient or Advanced on MAP Assessments (i.e., MAP Proficiency Rate)

2010 2011 2012 Three-YearCumulative

Student Gap Group, English/Language Arts

27.5% 22.8% 20.9% 24.0%

Student Gap Group, Mathematics

29.0% 26.4% 29.7% 28.3%

Combined English/Language Arts & Mathematics Proficiency Rate = 26.2%

Targets: Communication Arts = 44.21%; Math = 44.78%

38

And, by the way . . . The tests, they are a changin’

• In 2014-15, new state assessments will become operational in Missouri—assessments that are referenced to the Common Core State Standards and are designed to measure students’ readiness for college and careers.

• Two consortia are developing such assessments: * Smarter Balanced* PARCC

39

In closing . . . .

Questions & Comments . . . .

For further information, go to these websites.

MAP Assessments & MSIP: www.dese.mo.govAssessment Consortia: www.smarterbalanced.orgwww.parcconline.org

40