Children First Focus on Accountability Accountability Initiative July 19, 2006 CONFIDENTIAL.

Children FirstFocus on Accountability

Accountability Initiative

July 19, 2006

CONFIDENTIAL

2

• Accountability Initiative

–Vision and Overview

–Achievement Reporting and Innovation

System (ARIS)

• Accountability Initiative Implementation Timeline

CONTENTS

3

THE PROBLEM

Grade 4 ELA - 2001 to 2004 School Average

600.0

620.0

640.0

660.0

680.0

700.0

720.0

0 10 20 30 40 50 60 70 80 90

Mix of Free Lunch / Black or Hispanic /ELL/ Spec. Ed.

Avg

. Tes

t S

core

Number of Schools = 52

Number of Schools = 82

Number of Schools = 113

Number of Schools = 189

Number of Schools = 205

4

THE SOLUTION: A CYCLE OF CONTINUOUS IMPROVEMENT

Leadership / clear goals

Innovation / evaluation

Adjustment

SCHOOLSEnablers / Tools

• Continuous learning and support (e.g. Empowerment Schools Intensive)

• Periodic assessments

• ARIS

Accountabilities / Measures

• Progress Reports

• Quality Reviews

5

INNOVATIONS IN ACCOUNTABILITY

New accountability measurementsOld accountability measurements

• Snapshot of performance

• Movement only across single proficiency drivers- ignores outcomes on high/ low ends

• Only grades schools with challenging populations: SINI, SURR

• Incentives to focus on students least in need

• Progress: what the school adds

• All amounts of improvement in student achievement

• Universal application: All schools are graded: A, B, C, D, F

• Extra credit for gains by students most in need

• School improvement plan and target setting• Leadership change• School restructuring• School closure

4 Year Cycle of Consequences

Scatter Plot of Peer Group Index Scores with Overall School Index Scores with and without Extra Credit

0.000

0.200

0.400

0.600

0.800

1.000

1.200

1.400

0 10 20 30 40 50 60 70 80 90

Peer Group Index Score

Sc

ho

ol I

nd

ex

Sc

ore

SCORETOTNOEC

SCORETOTEC1

Linear (SCORETOTNOEC)

Linear (SCORETOTEC1)

ACCOUNTABILITIES / MEASURES: SCATTER CHART

7

ACCOUNTABILITIES / MEASURES: QUALITY REVIEWS

Gather data

Plan andset goals

Aligninstruction

Build and align capacity

Monitor and revise

Continuous data-driven improvement

• Understand what each student knows and is able to do monitor, and

• Measure the student’s progress over time.

• Set each student’s next learning step.

• Collaborate with student / parents to set goals for improving teaching practice and accelerating student learning

• Align school instructional activity and resources around focused plans for accelerating individual student learning

• Align instructional skills and capacity development around established goals for accelerating individual student learning

• Update documentation / best practices on effectiveness of its improvement plans and flexibility of change processes

8

Identify high leverage points for intervention

to improve student

performance

Align instructional practice and resources,

access external resources to

support change

Assess progress and make mid-

course corrections

Start by analyzing school’s performance data to ID areas that

need improvement

Create urgency

with publicized

results

Select a point of focusAnalyze student-

level data, instructional

practices, and adult learning

needs

ENABLERS / TOOLS: CREATE A CYCLE OF CONTINUOUS LEARNING

Top-down qualitative analysis from Quality Review

Bottom-up quantitative view from Assessments

Publish Progress Reports

Deeplyinformed

interventions

9

ENABLERS / TOOLS: A SET OF DIAGNOSTIC TOOLS-PERIODIC PROGRESS MEASURES

10

ENABLERS / TOOLS: DIAGNOSTIC ASSESSMENT METHODOLOGIES

October 2006 Jessica Holloway

Edward Smith

Raul Fuentes

Summarizing

Main Idea

Fact vs. Opinion

Predicting

Context Clues

Paraphrasing

December 2006 Jessica Holloway

Edward Smith

Raul Fuentes

Summarizing

Main Idea

Fact vs. Opinion

Predicting

Context Clues

Paraphrasing

11

ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION

Periodic assessments

DOE and vendor provided assessments

Default system based on citywide ELA and Math curricula • Math #1• Math #2

• Math #7

• ELA #1

• ELA #5

School customized assessments

School created assessments

Customized version provided to align with different curriculum, sequence or desired outcome (more predictive)

Standards-based alternative assessment system that..• Is developed by

school, network or region in collaboration with a partner

• Ensures equal rigor, measures progress and allows for cross-group comparison

At every grade level, a set of diagnostic assessment tools to help inform interventions and instruction to ensure an individualized instructional plan for every student

12

ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION

DOE and vendor provided assessments

Default system based on citywide ELA and Math curricula • Math #1• Math #2

• Math #7

• ELA #1

• ELA #5

School customized assessments

School created assessments

Customized version provided to align with different curriculum, sequence or desired outcome (more predictive)

Standards-based alternative assessment system that..• Is developed by

school, network or region in collaboration with a partner

• Ensures equal rigor, measures progress and allows for cross-group comparison

ARIS

13

• Accountability Initiative

–Vision and Overview –Achievement Reporting and

Innovation System (ARIS)

• Accountability Initiative Implementation Timeline

CONTENTS

14

OBJECTIVES AND SCOPE OF ARIS

Objectives

Ensure access to the information and tools necessary to enable

• Longitudinal and detailed analysis and reporting of achievement and performance data

• Best-in-class assessment design and implementation processes to extract data from multiple sources, including designing own reports and the capability to slice and dice the data

• Tools to improve learning of students with a range of identified needs;

• Sharing best practices and collaborating across schools to encourage cultures of continuous school improvement and professional learning

Scope of the ARIS

• Provide principals, teachers and parents with online information on student achievement, including periodic assessments

• Support development of longitudinal and detailed analysis

• Develop an integrated portal, including a dashboard to analyze key environment factors and achievement metrics

• Enable real-time prediction of school performance against year-end targets

• Generate standard reports for specific end-users

• Develop scorecards to enable drill-downs by student, assessment, strand and sub-strand analysis

• Provide knowledge management tools that capture teaching and assessment content generated at school level with an ability to approve, publish and share with the networks or the broader district. As well as capturing new quality review data

15

WHAT ARIS IS AND WHAT IT IS NOT

ARIS is… ARIS is not…

• A solution that…– Analyzes and reports– Assesses– Innovates (assessments, responsive

interventions)– Collaborates and personalizes– Shares knowledge (sharing) and

manages content

• A solution that may not be available off-the-shelf

• A solution that may require a partnership or consortium of Bidders to offer, not by a single vendor or a service provider

• A collaborative effort between the Vendor and NYCDOE to innovate, develop and execute

• A system that will replace all of the current applications now in operation at the DOE (e.g. ATS, HSST). Users will still need to update and print reports very much in the same way they do today but the access to all of the key systems will now all be well integrated through a single interface

• An Enterprise Data Warehouse (EDW) that will have all the current available data in DOE (assessment, Financial, HR, etc.). ARIS may be the foundation for an EDW in the long run, but initially the data warehouse will only contain student achievement data and the necessary information to analyze by several key criteria (e.g., age, school, grade, background, etc.)

• An opportunity to replace desktops and/or network infrastructure

16

OVERVIEW OF REQUIRED FUNCTIONALITY

Generate reports and analyze achievement/performance data

Use best-in class assessments to improve student achievement

Innovate the professional learning / share best practices across all schools

Analysis + Reporting• Canned and customized reports• Drill-down, roll-up, filtering• Ad-hoc queries• User-generated databases• Adding new data to central

repository• Data viewing in 3rd party tools

(e.g., Excel)

Advanced analysis (for selected users)• Advanced statistics• Pattern detection for target setting

Knowledge and plan management

• Library of templates for different document types (e.g., lesson plans, curriculums)

• Document creation and upload to repository

• Document tagging

• Collaborative document creation

• Publishing approval

• Document linking to reporting (surveys)

Knowledge and plan management (continued)• Searching documents in the repositories• Document viewing and download• Pattern analysis across text documents

Collaboration• Posting announcements; setting alerts• Online discussions; online messaging• Custom workspaces, integration w/email

Periodic assessments• For school- or center-created

– Library of templates/idea bank for school-created tests (e.g., rubric-based, standardized multiple choice)

– Test creation and upload• For 3rd party-created

– Assessment content upload into online testing, reporting and analysis

• For all– Online testing and scoring– Score data upload into

reporting and analysis

Personalization• Pre-defined home page based on user role• Personalization of home page (HP) by user• Adding new tools to HP from a tool catalog

Auditing (ie.,usage instances, frequency, modification history tracking)

1 2 3

17

KEY TECHNICAL AND SYSTEM REQUIREMENTS

Implications for ARIS RFP / requirementsContext and challenges• Data issues

– Significant archive size– Uncertain data quality– Critical data element gaps

• Legacy applications– Significant number, fragmented

functionality– Diverse but emerging

preferences for platforms– Goal of having a single

Enterprise Data Warehouse

• Infrastructure– Uneven / unclear quality of...

• Desktop standards / availability

• Network access– Stringent performance

requirements– Funding constraints

• Data issuesVendors, based on their experience, to work with DOE on developing a processes and tools for…– Data cleaning and migration– Filling critical data element gaps– Improving data entry quality

• Legacy applicationsBidders to..– Develop new data sources for new

accountability functions (e.g., QR)– Offer a scalable and evolvable data model– Comply with strongly preferred, but not

mandatory standards

• Infrastructure– Bidders have the option to host externally, or

suggest hardware for DOE-hosted solutions– Optimized for mobile/ existing infrastructure– Funding for infrastructure outside of ARIS

contract scope

18

DOE ARIS PREPAREDNESS TASK FORCE FOR KEY TECHNICAL AND SYSTEM REQUIREMENTS

Context

Objectives

• Data Integrity: Create solutions and define processes to ensure and maintain confidence in the quality of data and validity of ARIS outputs

• Infrastructure: Define the infrastructural improvements for maximizing usage and effectiveness of ARIS

• Data Source Inventory: Identify alternate types of approaches/ applications schools use to host data

• Communications: Craft messaging tailored to user experience

NYCDOE has set up a Task Force to set ARIS for success by preparing internal DOE systems and processes to complement vendor solution

The task force will…

• Identify level of cleanliness of existing data1

• Identify key causes for data inaccuracy or incompleteness

• Design and pilot solutions to clean existing source systems feeding ARIS2

• Recommend processes for maintaining data quality going forward

• Concentrate on transactional systems (ATS, HSST, etc.)

19

OVERVIEW OF TRAINING APPROACH

1Includes select admin staff, select teachers2Includes 25 Central Staff/Accountability Office; 10 power users drawn from schools3Includes 150 network/region; 4200 principals and other school achievement leaders4Primarily teachers

• Central/DAA (50)• Network / Regions (280)• Principals (1,400)• Other school achievement leaders1

(5,600)• Teachers (87,000)• Support staff / DIIT (?)

• Parents (~100s K)

• Initial training (role-based, project- and site-based)– One-on-one (for power and advanced)– Train-the-trainer (for standard) - school

achievement leaders train other teachers

• Ongoing need-based training– Phone support service– Online simulation exercises

• Initial training– Train-the-trainer – select

volunteers to train others

• Ongoing need-based training– Phone support service– Online simulation exercises

Phase 1 (Sep. 2007)

Phase 2 (Sep. 2008)

Usersegments

Trainingapproach

Role-based view

Skill-based view

( _ ) Segment size

• Power (352)• Advanced (~4,3503)• Standard (~90,0004)

• Standard (~ 100s K)

RFP requires the vendors to submit plans for measuring training effectiveness

20

• Accountability Initiative

–Vision and Overview

–Achievement Reporting and Innovation

System (ARIS)

• Accountability Initiative Implementation Timeline

CONTENTS

21

ACCOUNTABILITY INITIATIVE IMPLEMENTATION TIMELINE

2006-07 2007-08

Empowerment Schools

Other Schools All Schools

Progress Report (Previous Year) Yes No Yes

Progress Report (Current Year) Yes Yes Yes

School Targets Yes No Yes

School Grade Yes Yes Yes

Quality Review Yes Yes Yes

Rewards/ Consequences Yes No Yes

Data Management System Spring 2007 No Yes

Periodic Assessments Yes No Yes

Support for Continuous Learning Yes No Yes