Pilot of State Model Principal Evaluation System Year One Pilot of S.B. 191 Colorado Department of Education Educator Effectiveness September 12, 2012.
Post on 25-Dec-2015
218 Views
Preview:
Transcript
Pilot of State Model Principal Evaluation System
Year One Pilot of S.B. 191Colorado Department of Education
Educator Effectiveness
September 12, 2012
Evaluation and Continuous Improvement of the Statewide System to Evaluate the Effectiveness of Licensed Personnel
• The first year of the pilot went very well. While we have some interesting findings that we are monitoring, we do not have any recommended rule changes at this time. We believe the rules instituted in November are still appropriate and relevant for the upcoming year.
• Section 6.05 of the Rules for Implementation of S.B. 191:
The Department shall use information obtained through monitoring and reporting efforts to identify opportunities for improvement. No later than July 1 of each year, beginning in 2012, the State Board shall review these rules (1 CCR 301-87) and, informed by recommendations from the State Council and using information from implementation of the State Model System and other local systems, shall determine whether to affirm or revise the rules in order to reflect what has been learned.
Year One 2011-12 Development
and Beta Testing
• CDE ACTIVITIES• Develop State Model
Systems for teachers and principals
• Beta-testing of rubrics and tools
• Develop technical guidelines on Prof Practices and Student Growth
• Provide differentiated support for districts
• Populate and launch online Resource Bank
• Develop state data collection and monitoring system
• Develop tools for district implementation of system
Year Two 2012-13 Pilot and Rollout
• CDE ACTIVITIES• Usability study of
rubrics• Support pilot
districts through resources, training, tools, etc.
• Convene pilot districts to share lessons learned
• Analyze pilot district data and make adjustments as needed
• Train ALL non-pilot districts that are using the state model
• Make Recommendations on other licensed personnel (OLP) to State Board of Education (SBE)
Year Three 2013-14 Pilot and
Rollout
• CDE ACTIVITIES• Statewide
assistance on rollout of evaluation systems
• Develop evaluation system for other licensed personnel
• Support all districts through resources, trainings, tools, etc.
• Convene pilot districts to share lessons learned
• Analyze state data and make adjustments to the system as needed
• Validate teacher and principal rubrics
• Dev. criteria for evaluation training courses for approval by CDE
Year Four 2014-15 Full Statewide Implementation
• CDE ACTIVITIES• Finalize statewide
implementation of teacher/principal systems
• Pilot OLP rubrics• Continue support to
districts via resources and training
• Ensure there are evaluator training courses throughout the state
• Analyze data and make adjustments as needed
• Make recommendations to SBE this year and all following years for Continuous Improvement
Timeline of Implementation
• Piloted in 27 districts across the state
• Principal pilot data received from 22 districts
Principal Evaluation System
CDE Pilot District CDE/CLF Integration District
1 Center 1 Centennial2 Crowley 2 Archuleta 3 Custer 3 Bayfield4 Del Norte 4 Dolores RE-25 Eads 5 Dolores RE-46 Jefferson 6 Durango7 Miami-Yoder 7 Eagle8 Moffat 8 Ignacio9 Mountain Valley 9 Mancos
10 Platte Canyon 10 Montezuma Cortez11 Salida 11 Silverton12 South Routt 12 Thompson13 St. Vrain14 Valley Sterling15 Wray
# of evaluations expected*
# of evaluations
received % received
Statewide 526 241 46%
Range 0-298 0-131 0-200%* total # of administrators
Overall Reflections on the Pilot
• Managing change• Trainings of Superintendents, Principals, and Assistant Principals
throughout the fall of 2011• “Finally we have a road map”• Learning about management of the process• Interest and curiosity on Student Growth• Overall, we are getting positive response
Quality Standard 1: Principals demonstrate strategic leadership. – Element a: School Vision, Mission and Strategic Goals– Element b: School Plan– Element c: Leading Change– Element d: Distributive Leadership
Quality Standard 2: Principals demonstrate instructional leadership. – Element a: Curriculum, Instruction, Learning and
Assessment– Element b: Instructional Time – Element c: Implementing High-quality Instruction – Element d: High Expectations for all Students – Element e: Instructional Practices
Quality Standard 3: Principals demonstrate school culture and equity leadership.
– Element a: Intentional and Collaborative School Culture – Element b: Commitment to the Whole Child– Element c: Equity Pedagogy – Element d: Efficacy, Empowerment and a Culture of
Continuous Improvement
Principal Quality Standards and Elements
Quality Standard 4: Principals demonstrate human resource leadership. – Element a: Professional Development/Learning Communities– Element b: Recruiting, Hiring, Placing, Mentoring, and
Dismissal of Staff– Element c: Teacher and Staff Evaluation
Quality Standard 5: Principals demonstrate managerial leadership. – Element a: School Resources and Budget – Element b: Conflict Management and Resolution– Element c: Systematic Communication – Element d: School-wide Expectations for Students and Staff– Element e: Supporting Policies and Agreements– Element f: Ensuring an Orderly and Supportive Environment
Quality Standard 6 : Principals demonstrate external development leadership.
– Element a: Family and Community Involvement and Outreach
– Element b: Professional Leadership Responsibilities – Element c: Advocacy for the School.
The italicized elements were not included in the 2011-12 pilot of the principal evaluation system but will be included in the 2012-13 pilot.
Ratings on Quality Standards
• Timeframe of the cumulative evaluation was October 2011 – May 2012.
• Principal evaluators were instructed to conduct a mid-year review and an end-of-year evaluation (end-of-year evaluation ratings are reported in this presentation).
• Individual districts determine how many (or if) observations were conducted.
• Evaluation ratings include observable and non-observable components of Quality Standards 1-6, for an overall assessment of Principal performance.
Standard 1
Standard 2
Standard 3
Standard 4
Standard 5
Standard 6
Overall Rating
0
50
100
150
200
250
ExemplaryAccomplishedProficientPartially ProficientNot Evident
Num
ber o
f Adm
inist
rato
rs
Ratings Distributions: Standards Summary and Overall Rating
Standard 1 – Strategic LeadershipStandard 2 – Instructional LeadershipStandard 3 – School Culture and Equity LeadershipStandard 4 – Human Resource LeadershipStandard 5 – Managerial LeadershipStandard 6 – External Development Leadership
• 90% of principals received a rating of Proficient or above on the Overall Rating.• Principals received higher ratings on Standard 4 (HR Leadership) and lower ratings on Standard 1 (Strategic
Leadership).
Element 1a Element 1b Element 1c Element 1d Standard 10
50
100
150
200
250
ExemplaryAccomplishedProficientPartially ProficientNot Evident
Num
ber o
f Adm
inist
rato
rs
• Strategic Leadership is the lowest rated Standard, with 86% of educators given a rating of Proficient or higher.
Element 1a – School Vision, Mission and Strategic GoalsElement 1b – School Improvement PlanningElement 1c – Leading ChangeElement 1d – Distributive Leadership
Standard 1 – Strategic Leadership
Element 2a Element 2b Element 2c Element 2d Standard 20
50
100
150
200
250
ExemplaryAccomplishedProficientPartially ProficientNot Evident
Num
ber o
f Adm
inist
rato
rs
• Instructional Leadership contains the highest rated Element (Element 2b), with 95% of educators receiving a rating of Proficient or higher.
• This Standard also contains the lowest rated Element (Element 2c), with 69% of educators receiving a rating of Proficient or higher.
Element 2a – Curriculum, Instruction, Learning and AssessmentElement 2b – Instructional TimeElement 2c – Implementing High Quality InstructionElement 2d – High Expectations for All Students
Standard 2 – Instructional Leadership
N Standard 1 Standard 2 Standard 3 Standard 4 Standard 5 Standard 6
Overall Rating
CSAP Reading MGP 211 .316** .294** .278** .224** .271** .095 .309**
CSAP Math MGP 211 .262** .221** .237** .175* .192** .053 .273**
CSAP Reading % of students proficient or advanced
213 .134 .071 .132 .106 .191** .127 .133
CSAP Math % of students proficient or advanced
213 .202** .178** .196** .161* .190** .115 .160*
** Correlation is significant at the 0.01 level* Correlation is significant at the 0.05 level
• Correlations indicate the strength of the relationship between two measures (in this case, between evaluation ratings and aggregate CSAP scores). 0 indicates no relationship and 1 indicates a perfect positive relationship. Most of the correlations below are small but statistically significant.
• On average, principals who received higher evaluation ratings also have higher student growth and achievement in their school. Principal ratings were positively correlated with:
– CSAP Reading growth – CSAP Math growth and achievement
Correlations with Students’ Academic Outcomes
Student Growth
StudentAchieve-ment
• Fairness - the extent to which a system is marked by impartiality and free from favoritism– Rubric ratings don’t vary based on student, school, or principal demographic characteristics
(e.g., don’t vary based on the % of minority students in a school or the principal’s gender). These findings indicate that the rubric reflects a common standard that is fair and equally
applicable across different demographic groups.
• Validity - the extent to which a variable, scale, or set of measures accurately represents the concept of interest– Rubric text is aligned with other content and leadership standards.– Element ratings are internally cohesive and will group into established Standards.– Rubric ratings are correlated with other measures of school success, specifically the aggregated
CSAP growth and achievement in the school.– Rubric ratings don’t vary based on student or school characteristics, indicating that the rubric is
valid across different demographic groups. These findings indicate that the rubric is measuring the right things, we’re measuring what we
think we’re measuring, and what we’re measuring is aligned with other measures of success.
Investigating Whether the System is Fair and Valid
• Reliability - the extent to which a variable or set of variables is consistent in what it is intended to measure– Rubric ratings at the Standard level are highly correlated with each other.– Rubric ratings at the Element level are highly correlated with each other.– Elements within Standards are highly correlated with each other.– Strong reliability coefficients for each Standard.
These findings indicate a consistency of ratings across Standards, across Elements, and across Elements within their designated Standards. For example, principals who received a rating of Exemplary on Standard 1 were more likely to receive Exemplary ratings on the other Standards.
These findings indicate that each Standard captures a dimension of school leadership.
Investigating Whether the System is Reliable
Principal Baseline and Feedback Surveys
• Principals were surveyed before (i.e., baseline) and after (i.e., feedback) they experienced the state model system in the 2011-12 school year.
• Across the board, principals responded more positively in the post-survey to questions about the state model system.
0
20
40
60
80
100
My current system…The state model system…
% o
f Edu
cato
rs G
ivin
g a
Posi
tive
Res
pons
e
Next Steps in the Pilot of the State Model Evaluation System
• Run additional analyses on 2011-12 principal pilot data– Example: Compare with data from the Teaching, Empowering, Leading and
Learning (TELL) Survey (TELL is the state’s biennial educator perception survey)
• Continue to pilot the principal evaluation system in the 2012-13 school year
• Pilot the teacher evaluation system in the 2012-13 school year• Build out the student growth component• Focus on inter-rater agreement, which involves continued training
and calibration of evaluators• Continue to collect rubric and other evaluation data in order to
conduct studies of reliability and validity
top related