Csde final class ppt 1

1

An Analysis of Quality Review Alignment with Selected

Governance Models

Connecticut Quality Review TeamJamie Alter, Gabriella Barbosa, Ranjana Reddy, Ryan Thorpe, William David Williams

April 8, 2013

2

Agenda

• Overview of Connecticut Quality Review project

• Design choices made to align Connecticut Quality Review with experimentalism

• Challenges in designing an experimentalist Quality Review system

• Examination of an unresolved question: Should CSDE use Quality Review for school accountability?

3

Created first iteration of QR standards, tools, and process based on research and effective practices

The team supported the Connecticut State Department of Education by designing standards, tools, and a strategic process for piloting Quality Review.

Conducted focus groups with CSDE to solicit feedback

Strategically recruited districts to participate in pilot

Solicited feedback on QR from district leader

Designed Reviewer and School Guides

Supported training of reviewers and orientation for schools and district personnel

2. Develop and implement a rigorous

instructional program that meets the learning needs

of all students

• High quality schools evaluate and select appropriate strategies, plans, instructional programs, and curriculum for their circumstances to support college and career readiness for all students.

• Similarly, they organize themselves so that school leaders are helping to facilitate continuous improvement in educator effectiveness at meeting learning goals.

3. Create a school culture and climate that enables

all students to learn

• High quality schools promote a positive learning environment and engage families and the community in continuous improvement.

Rationale

• High quality schools have an overall vision focused on a pathway to college and career readiness for all students that has generated a thoughtful theory of action, comprehensive data gathering, and careful plans for implementing the theory of change.

1. Create and monitor a vision and theory of

action focused on student learning

4. Align management and operations to facilitate

achievement of student learning goals and steadily improve

• High quality schools continuously monitor and improve management practices to ensure the are effectively serving student learning as the school’s primary objective

During the QR process, reviewers will assess schools against four overarching standards.

4

Overview of Standard

Standard Indicator (To what extent do schools…)

Student Learning-Centered Culture

Continuously-Improving Instruction

1.1 Create a Shared Vision for Student Learning

1.2 Develop Theory of Action and Goals

1.3 Target Student Needs

1.4 Assess and Monitor Progress

2.1 Develop Rigorous Curriculum

2.2 Build Instructional Capacity

2.3 Develop Teacher Leadership

2.4 Promote Evidence-based Instruction

These standards are aligned to 17 indicators of school practice.

Standard Indicator (To what extent do schools…)

Student and Family Investment

Operational Alignment to

Goals

3.1 Establish Welcoming Environment

3.2 Engage Teachers and Staff

3.4 Communicate with and Engage Families

4.1 Align Resources and Policies to Goals

4.2 Monitor and Revise Operations

3.5 Build Community and Family Partnerships

4.3 Manage Talent

4.4 Develop and Pursue Innovative Solutions

3.3 Engage Students

These standards are aligned to 17 indicators of school practice.

The Quality Review process involves pre-review data collection, the 2-day onsite review, and post-review feedback.

Pre-review Review Post-review

Description:

Key steps:

Reviewers and schools prepare for the formal process of data collection by gathering documents, examining school data, self-evaluating, and setting logistical parameters.

During a two-day school visit, reviewers collect and synthesize evidence from a variety of sources to gain a full picture of school performance against the set of standards for effective schools.

Reviewers provide verbal and written feedback to schools based on evidence collected during the quality review and facilitate a structured debrief and reflection on results.

• Principal Orientation• School Self-Evaluation• Conversation with District

Staff• Planning Conversation Among

Reviewers• Planning Conversation with

School Leader

• Classroom Observations• Constituent Interviews• Observation of Teacher Teams• Reviewer Check-ins• Principal Debriefs

• Presentation of Findings• Goal Setting and Planning • Rating and Scoring• Written Report• Verification• Reconsideration, if necessary

The pilot will include two teams of reviewers conducting quality reviews in 4 East Hartford schools in April/May.

Reviewer Team 1

Reviewer Team 2

School 1

School 2

School 3

School 4

REVIEWER TEAMS• Include at least 1

state and 1 district reviewer

• Trained on the QR standards and process by CSDE vendor

• Have experience as educators and/or educational leaders

SCHOOLS

• Four elementary schools; identified by CSDE as Focus Schools

• Located in Alliance District

• District has 61% FRL, 47% proficient reading, 64% proficient math (Grade 3 CMT)

9

The QR pilot is designed to accomplish three key objectives.

To increase stakeholder support for Quality Review

• East Hartford will have an opportunity to test-drive quality review. • The CSDE will conduct outreach sessions with other Alliance Districts to

encourage more districts to engage in Quality Review in future years.

To troubleshoot and improve the

standards, tools, and process

• Reviewers, districts, and schools will provide feedback on the pilot tools, process, and reports.• This feedback will be used to revise the QR system and to inform

recommendations for roll-out.

To build capacity within the state

• State reviewers will be trained to conduct reviews, evaluate schools, write reports, provide verbal feedback to schools, and to ensure inter-rater reliability through internal training.• The CSDE will develop internal procedures for collecting and analyzing data

from quality reviews and managing the pre- and post- review elements of the process.• Personnel from East Hartford will gain experience as co-reviewers.

10

Agenda





11

CT QR was designed to inform an “experimentalist” problem-solving process at the school level, by providing schools with targeted feedback on areas of improvement

Enhance Strategic Feedback and Learning

Communicate and Link Strategic Objectives

and Measures

Plan, Set Targets and Align Strategic

Initiatives

Clarify and Translate Visions and Strategy

SCHOOL PROBLEM

SOLVING FOR IMPROVEMENT

CT QR will ensure that expertise flows from local learning to the center, and that the state, in turn, will accumulate and share local learning across schools.

State Knowledge Base of Effective

Practices

SCHOOL SCHOOL

SCHOOLSCHOOL

13

Other specific design features of CT QR were also intentionally linked to experimentalism.

How it Meets an Experimentalist ObjectiveFeature of CT QR

Strategic, Open-Ended Standards

The system is built around standards that exemplify effective schools, while being broad enough to generate fruitful conversation and learning among schools operating in different contexts.

Involvement of Other Stakeholders

Involvement of School Throughout Process

Processes such as self-evaluation, principal check-in, and goal setting meetings empower local school leaders to be a critical part of monitoring and adjusting their own performance.

Experimentalism allows for a diverse group of stakeholders to be included as part of an accountability system, including those usually left out of the conversation like parents and the community.

14

Minimal or no evidence

• Minimal process. School does not have in place any coherent system for regularly monitoring the effects of school operations and management systems on student learning outcomes.

Progressing toward standard

• Limited process. School has a plan for monitoring effectiveness and efficiency of operations and management systems for their effects on student learning outcomes, but inconsistently uses the information learned from this monitoring.

Meets standard

• Strategic process. School effectively and strategically plans and executes systems to regularly monitor and revise effectiveness and efficiency of operations and management systems to improve their impact on student learning outcomes.

Exceeds standard

• Strategic process and continuous improvement. School effectively and strategically plans and executes systems to regularly monitor and revise effectiveness and efficiency of operations and management systems to improve their impact on student learning outcomes.

Standard 4: Operational Alignment to Goals Indicator 4.2: Monitor and Revise Operations

15








Experimentalism allows for a diverse group of stakeholders to be included as part of an accountability system, including those usually left out of the conversation like parents and the community.

16

Self-Review ensures the QR process is collaborative, and empowers stakeholders by helping them engage in a structured process of reflection on strengths and weaknesses.

Reflect:• How are we doing?• How do we know?• How did we get here?

Plan:• What are our goals?• How will we achieve them?• What support do we need?

Execute:• How do we put our plan into

action?• What more needs to happen

to make the plan a reality?

Monitor:• What progress are we

making toward our goals?• How well are we aligning

resources to achieving our goals?

Collaborate:• With whom do we need to

partner to support our goals?• Who needs to know about

our goals and progress?

17








Experimentalism allows for a diverse group of stakeholders to be included as part of an accountability system, including those usually left out of the conversation like parents and students.

18

Constituent conversations allow reviewers to gain information about stakeholders’ perspectives and experiences with the school across multiple indicators.

Focus Group with Teachers

Focus Group with Parents

Focus Group with Students

19

Agenda





Tension #1: An experimentalist QR relies on autonomy at the school level, but autonomy is noticeably constrained.

RESOLUTION

By working with the state and districts to design and implement QR, we ultimately hope to build buy-in for necessary structural changes at the school level. Essentially, states and districts will see that schools have internal capacity to drive change and that they need additional autonomy to facilitate improvement through the QR process.

EXPERIMENTALIST PROPOSAL

Schools should have the discretion to develop context-specific strategies to close gaps identified by the QR system

BUREAUCRATIC REALITY

Schools have limited flexibility in allocating financial resources and guiding talent management constrain their ability to make changes advocated by QR

21

Tension #1 Resolution: Indicator 4.3 Managing Talent was adapted to acknowledge school-level constraints around talent management.

Develop and implement a strategy for managing human capital, including identifying, recruiting, mentoring, providing career-enhancing opportunities for, and retaining the most effective teachers and assigning teachers to areas where their skills best match student learning needs.

Develop and implement a strategy for marshaling resources to manage human capital, including identifying, recruiting, mentoring, providing career-enhancing opportunities for, and retaining the most effective teachers and assigning teachers to areas where their skills best match student learning needs, given each individual school’s contextual constraints.

Original Indicator Adapted Indicator

Tension #2: An experimentalist QR uses broad standards that are not prescriptive, but stakeholders want clear expectations.

RESOLUTION

The version of QR being piloted includes a rubric with broad, experimentalist standards and an evidence document with narrative-style vignettes as

examples of what "well-developed" looks like for each indicator.

EXPERIMENTALIST PROPOSAL

CPRL proposed a rubric with intentionally broad standards

to encourage a dialogue between reviewers and school

personnel and to resist the tool’s use as a compliance

mechanism.

BUREAUCRATIC REALITY

The CSDE and many school and district personnel believe

that providing specific standards with concrete

expectations for acceptable evidence ensures the process is fair and best communicates expectations to stakeholders.

Tension #2 Resolution: Revised QR includes evidence document with vignette-like narratives to provide examples of “well-developed.”

Indicator 1.1Create a widely shared vision of the progress students are expected to make while at the school to get and stay on track to graduate on time and become college and career-ready and foster a deeply embedded culture of high expectations committed to realizing that vision.

“The school effectively conveys high expectations to students and parents that result in a deep commitment to the school’s vision. Success is celebrated during “Honors Night” where teachers, parents and students are rewarded for exemplary work. The school provides parents with continuous information on how well their children are learning via report cards, progress reports and Engrade, an on-line reporting system. In an effort to better support students and effectively work with families, the school administers a student survey that aids the school to better understand the circumstances of different families and differentiate support accordingly. Guidance counselors work closely with families and actively assist in the completion of high school applications ensuring that students and families make good choices when selecting high schools. The emphasis on high standards and open exchange of information results in an atmosphere of care and support. Consequently, parents report appreciating the high expectations for attendance, good citizenship and academic effort within a highly inclusive and welcoming environment.”

Original version included only indicators Adapted version added vignette-style narratives

24

Agenda





The CSDE faces a decision about whether to use the results of Quality Review as part of its system for school accountability.

Factor in School Accountability

Purely Developmental

School ratings on the qualitative indicators would be integrated with the quantitative metrics currently used (state test scores and graduation rates) to classify schools and to determine which schools require intervention.

Schools would continue to be classified based on only quantitative metrics; Quality Review ratings will be provided only to school and district leaders, who will be encouraged to use the information to drive their own improvement.

Professionalism

Experimentalism

Performance Managemen

t

Experimentalism

Performance management suggests using QR for Accountability whereas professionalism would use QR purely developmentally.

QR as a Factor in School

Accountability

QR as a Purely Developmental Tool

Teachers and school leaders…. QR should be…

The CSDE’s decision about how to use Quality Review is closely related to its theory of action for school improvement.

If the CSDE believes…

Schools are likely to respond to QR for stakes by…

• “gaming” the quality review by prepping for the review and performing on the day of the site visit

Factor in student

accountability

• taking the standards more seriously and working harder to authentically improve

• be fair, valid, include both leading and lagging indicators, and provide schools with actionable information

• help the state identify schools that need the most support

• are professionals who, when provided with information and autonomy, are motivated to improve without sanctions

Purely Developmental

Systems of school accountability should….

• are motivated by fair and valid assessments of performance that are public and result in sanctions and interventions

Then…

Discussion teams

Team 1 Dara, Michael, Rohan

Team 2

Team 3

Seo Yun, Tom, Zahreen, Jason

Team 4

Jill, Sana, Brian, Jason

Team 5

Regina, Alia, Matthew

New Haven

Discussion Role Play• Take a point of view on whether Quality Review should be

part of Connecticut’s school accountability system and why• If so, how would this work?

– Publically reported– Factor in classifying schools– Factor in determining whether schools are closed or turned

around

• Split into groups of three or four and take the following roles:– Person who is youngest will represent a Teacher in the School– Person who is second youngest will represent the Leader of the

School– Person who is third youngest will represent the District

Superintendent– Person who is the oldest will represent the State Commissioner

• Each party should advocate for the use quality review that is best from the assigned perspective.

• Note that each entity likely has a different viewpoint regarding the purpose for conducting of quality review as well as which use of quality review best serves the stated purpose.

Task

Roles

30

Meets primary objectives

Create a common language around school quality

Improve accountability system

Likely Unlikely

Build capacity in districts and state

As you discuss whether QR should be a factor in school accountability, consider which use of QR is most likely to meet each of QR’s primary objectives.

Build support among educators and the public

ExplanationPurely

Dev.Part of Acct.

Source: Interviews with districts and vendors.

Encourage and enable authentic school improvement

Report out.

Teacher

Principal

Superintendent

State Commissioner

• To come

Factor in Accountability? Purely Developmental?

• To come

• To come • To come



Appendix

33

NCLB exposed poor performance, but is blunt and limited; Connecticut is developing a richer, more nuanced system.

NCLBUse status-based accountability and

identification of “schools in need of improvement” to motivate schools to

improve

Impose sanctions if schools do not improve

Use rich accountability and tiered classification to enable

and motivate schools to improve

Support low performing schools in the process of strategic

planning and intervention

Give targeted low performing schools the flexibility and

resources they need to improve

Provide all schools with tools and information needed to

facilitate improvement

Impose sanctions if schools do not improve

A richer system

34

Qualitative Review could be integrated into Connecticut’s new accountability system for all schools.

Quantitative Factors

SPI < 6464 < SPI < 88; did not meet

annual targets

64 < SPI < 88;met annual

targetsSPI > 88;

met state goals

Qualitative Factors

Well Developed Transition Progressing Excelling Excelling

Proficient Transition Transition Progressing Excelling

Developing Review Transition Transition Progressing

Under-developed Review Review Transition Transition

35

Qualitative Review could be integrated into Connecticut’s new accountability system as an additional component for schools on the cusp.

Excelling

Progressing

Transition

Review

Schools on the cusp of Excelling

Schools on the cusp of Progressing

Schools on the cusp of Transition

Schools receive QR for next three years

If score Proficient or Well-Developed, will be classified in the higher category even if quantitative data falls short by slight margin.

36

Qualitative Review could be integrated into Connecticut’s new accountability system as an additional requirement for exiting Focus and Turnaround status.

Focus Status

Turnaround Status

Lowest subgroup SPI or grad rates

Lowest SPI for “all students”

Meet subgroup SPI and grad rate targetsAchieve Proficient or Well-Developed QR Score

Meet SPI and grad rate targetsAchieve Proficient or Well-Developed QR Score

Csde final class ppt 1

Education

quality review process

schools collection

focus school

schools prepareduring

school performance

cmt school

school accountability

school leadersstudents