Evaluation Systems: Context, Issues, and Perspectives Charles Igel Senior Researcher, McREL Brian Ewert Superintendent, Englewood Public Schools Barbara Lunsford Associate Superintendent, Georgia DOE Kerry Englert President, Seneca Consulting Jessica Allen Researcher, McREL
40
Embed
Evaluation Systems: Context, Issues, and Perspectives Charles Igel Senior Researcher, McREL Brian Ewert Superintendent, Englewood Public Schools Barbara.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Evaluation Systems: Context, Issues, and Perspectives
Charles Igel Senior Researcher, McREL
Brian Ewert Superintendent, Englewood Public Schools
Barbara Lunsford Associate Superintendent, Georgia DOE
Kerry Englert President, Seneca Consulting
Jessica Allen Researcher, McREL
Tony Davis Senior Director, McREL
Framework for Evaluation Systems
Teacher Effectiveness
Quality Standards
Leadership Environment Content Learning Practice Growth
50%
50%
Performance Ratings
Developing Proficient Accomplished Distinguished
Professional Practice
Examples of large-scale evaluation initiatives
Considerations for developing and rolling-out educator evaluation systems
Colorado SCEE Example
Human Judgment: Data should inform decisions, but human judgment will always be an essential component of evaluations
Continuous Improvement: The implementation and assessment of the evaluation system must embody continuous improvement
Feedback: The purpose of the system is to provide meaningful and credible feedback that improves performance
Alignment: Educator evaluation must take place within a larger system that is aligned and supportive
State Council for Educator Effectiveness. (2011, May). State Council for Educator Effectiveness report and recommendations. Denver, CO. Available at www.cde.state.co.us/EducatorEffectiveness
S1: Facilitating the development and implementation of a shared vision for learning
S2: Nurturing a culture conducive to student learning and professional growth
S3: Managing for a safe, efficient, and effective learning environment
S4: Collaborating with and mobilizing faculty and community resources
S5: Acting in a fair and ethical mannerS6: Responding to and influencing the political, legal, and cultural
environment
Additional considerations: InTASC Model Core Teaching StandardsS1: Uses of developmentally appropriate instructionS2: Establishes inclusive learning environmentsS3: Promotes positive peer interactions and engagement with materialS4: Understands content and makes it meaningful and accessible to all
learnersS5: Connects content in ways that promote creativity and critical
thinkingS6: Uses multiple methods of assessmentS7: Uses all professional knowledge in lesson planningS8: Uses multiple instructional strategies S9: Active in own professional growthS10: Take appropriate leadership positions and actively engages in the
profession
Additional considerations: Educational Data Exchange Network (EDEN) Reporting
What is reported?1) Teacher counts categorized by performance level 2) Total number of teacher evaluated3) Principal counts categorized by performance level4) Total number of principals evaluated
Guidance on –1) teachers and/or principals assigned to multiple schools in the same LEA2) teachers and/or principals assigned to multiple schools in different LEAs
• Deficit/Punitive Model vs. Continuous Improvement Model
• Lack of Performance Pathways for Growth
• Lack of Policies, Procedures, and Agreed Upon Practices
• Lack of a Framework to Drive Conversation
• Lack of Goal Setting and Professional Growth
The LifeboatThe Lifeboat
• Teacher/Principal Effectiveness Bill
• Creates a four-tiered system to evaluate the effectiveness of licensed personnel as a means of improving the quality of education in Colorado. The basic purposes of the statewide system are:
To ensure that all licensed personnel are evaluated using multiple, fair, transparent, timely, rigorous, and valid methods, fifty percent of which evaluation is determined by the academic growth of their students
To ensure that all licensed personnel receive adequate feedback and professional development support to provide them a meaningful opportunity to improve their effectiveness
To ensure that all licensed personnel are provided the means to share effective practices with other educators throughout the state
The LifeboatThe Lifeboat
• Solidarity & Alignment: Board, Administrators, Principals, & Union
End with an anecdote about the quality of the conversations, teachers making independent connections between indicator language and SIP's, bring in back to a snap shot of what it sounds like when it;s working well.
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Georgia Department of Education
6/19/2011 19
CLASS KeysClassroom Analysis of State Standards
A Teacher Evaluation Process Standards
Barbara LunsfordAssociate Superintendent
School Improvement
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
CLASS KeysSM Strands and Elements
6/19/2011 20
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
ASSESSMENT - The collecting and analyzing of student performance data to identify patterns of achievement and underachievement in order to design and implement appropriate instructional interventions.
STRAND STANDARD ELEMENT
CONTINUUM OF IMPROVEMENT RUBRIC
6/19/2011 21
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Not Evident Emerging Proficient Exemplary
6/19/2011 22
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org 6/19/2011 23
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Performance on the elements of the CLASS KeysSM is identified on a four-level continuum.
Not EvidentNot Evident EmergingEmerging ProficientProficient ExemplaryExemplary
This continuum is not utilized to label teachers as Not Evident, Emerging,
Proficient, or Exemplary.
The continuum is used to describe a teacher’s PERFORMANCE on
specific elements.
6/19/2011 24
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
GTDRPerformance
An array of evidence is collected from multiple sources during the year.
AnnualAnnualEvaluationEvaluation
Announced,Longer
Observations
Unannounced, Short
Observations
Other artifacts and evidence from
conferences, meetings, review
of teacher and student productsStudent Achievement Data
6/19/2011 25
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Module 1: Content and Structure (The What?)
Module 2: Overview of the Evaluation Process (The How?)
Module 3: Self-Assessment and Reflection
Module 4: Professional Growth Plan
Module 5: Pre-Evaluation Conference
Module 6: Informal Observations
Module 7: Formal Observations
Module 8: Georgia Teacher Duties and Responsibilities
Module 9: Annual Evaluation
Module 10: Professional Development Plan
CLASS KeysSM Modules
CLASS KeysSM
Modules
6/19/2011 26
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Evaluator Training Assessment
Three Sections:• Scoring the Elements in a Classroom Video• Annual Evaluation Performance Task• Technical Questions Regarding Process
6/19/2011 27
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org 6/19/2011 28
KERRY ENGLERT, PH.D.SENECA CONSULTING
JESSICA ALLEN, MAMcREL
COUNCIL OF CHIEF STATE SCHOOL OFFICERSNATIONAL CONFERENCE ON STUDENT ASSESSMENT
JUNE 19TH, 2011
Validity in a Standards-Based Teacher Evaluation
System
Presentation Purposes
Overview of our experiences conducting an initial validity study of a state’s standards-based teacher evaluation system. Propose a framework that can be used when
designing these validation studies. Offer suggestions and considerations for validation
studies of educator evaluation systems.
National Issues in Teacher Effectiveness
Education Research Connection between teacher effectiveness and student
learning that is well documented (ex. Goe (2007) ).
Education Policy Focusing on teacher evaluation and the relationship
between teacher effectiveness and student achievement. Race to the Top White House’s Blue Print for Reform
Standards-Based Evaluation System
Three core components: A set of standards. Procedures for collecting performance data on the
standards. Rubrics to evaluate performance.
Optional fourth component: Methodology for incorporating evaluation results into
personnel decisions.
* from (Odden, 2004):
Why do a validity study?
Race to the Top Requires a valid and reliable teacher evaluation
instruments.
Validity evidence Builds credibility to the instrument and evaluation
system. Provides information on data use and implementation
Provides information on continual development. Can focus training efforts.
CLASS KeysSM Teacher Evaluation System
Purposes That serves a two-fold purpose improvement and
accountability
Core Components Five major strands Training materials 28 performance rubrics.
Validity Study of CLASS KeysSM System
Study Context Quick turn around time Extant data from pilot study Readily available and accessible information
Broad examination of the literature Inform, focus, and guide the study Better capitalize on the available data
Relevant Literature
Validity Theory and Standards A unified theory and system uses (Standards for
Educational and Psychological Testing, 1999; Messick, 1989)
A strong theory of action (Kane, 2006) Personnel Evaluation
Applying the concepts of fairness, transparency, efficiency and practicality to the personnel evaluation context (Gullickson, 2008)
Quality Rubrics Characteristics of quality rubrics (Arter & McTighe, 2001)
Implementation Fidelity The degree to which the system is being implemented
according to the specification (Ruiz-Primo, 2006)
Potential framework for validity in Educator Evaluation
Validity in
Educator
Evaluation
Methods related to the framework
Validity Theory and Rubric Quality Content
Validity Theory Construct
Personnel Evaluation Standards and Validity Theory Fairness and Reliability
Implementation Fidelity and Validity Theory Use and Interpretation
Future Considerations
Validation of educator evaluation systems might benefit from multiple perspectives.
Pilot and field tests are opportunities not only to test the instrument but apply validity methodologies and instruments.
Validation is an ongoing process and provides a framework to understand how the instrument is being used.
References
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological tests and manuals. Washington, D.C.: American Psychological Association.
Arter, J., & McTighe, J. (2001). Scoring rubrics in the classroom: Using performance criteria for assessing and improving student performance. Thousand Oaks, CA: Corwin.
Goe, L. (2007). The link between teacher quality and student outcomes: A research synthesis. National Comprehensive Center for Teacher Quality. Retrieved from http://www.tqsource.org/publications/LinkBetweenTQandStudentOutcomes.pdf
Gullickson, A. R. (2008). The Personnel Evaluation Standards: How to Assess Systems for Evaluating Educators, 2nd edition. Joint Committee on the Standards for Educational Evaluation. Corwin Press, Thousand Oakes, CA.
Kane, M. T. (2006). Validity. In R. L. Brennen (Ed.), Educational Measurement (Vol. 4th edition). Westport, CT: Praeger Publishers.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13-103). New York: American Council on Education & McMillian.
Odden, A. (2004). Lessons learned about standards-based teacher evaluation systems. Peabody Journal of Education, 79(4), 126-137.
Ruiz-Primo, M. A. (2006). A multi-method and multi-source approach for studying fidelity of implementation. National Center for Research on Evaluation, Standards, and Student Testing (CRESST).