INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS
Feb 24, 2016
INTER-RATER AGREEMENT IN KANSASSummer Principals AcademyJuly 22-24, 2014Abilene, KS
2
What is Inter-Rater Agreement?• Consistent
understanding• Consistent
agreement• Consistent
process• Consistent
credibility
The degree 2 raters using the same scale give the same rating in identical situations.
3
Where is Inter-rater Agreement Useful??
Inter-Rater Agreement is useful in all areas of administrative practice including:
• Kansas Educator Evaluations• Kansas Accreditation • Personnel Decisions• Family Engagement Programs• Budget Management
4
What are the Goals of Inter-rater Agreement?
• The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart.
• The Goal is to have an administrative Common Frame of Reference for your district such as:• Agreement with no rating more than one level apart. • Consistent understanding• Consistent agreement• Consistent process• Consistent credibility
5
How is Inter-rater Agreement Helpful??
• What is a Common Understanding?
• How does Inter-rater Agreement help administrators??
• Facilitates Quality• Provide Consistent Messaging• Impacts Increased Student Growth• Enhances Trust • Facilitates Quality Feedback• Legally Defensible • Assists Accurate Data Collection• Allows for Increase Cooperation
over Time • Passes Public Scrutiny
6
District-wide Agreement about External Criteria
• Agreed Upon External Criteria for:• Observations• Artifacts• Units of Study• Processes• Program Development• Time Limits • Walk-Throughs• Protocols
7
Pragmatics/ConsiderationsPotential Landmines
• Discuss with an Elbow Partner Possible Issues that may cause Rater Error• Leniency: Benefit of the
Doubt• Central Tendency: Safety in
the Middle• Comparing One with
Another: Stick to the Rubric• Relying on Gut too Much:
Stick to the Rubric• Evaluating Absent Evidence:
Stick to What is Observed
8
How to Train Administrators in using Inter-rater Agreement
• Inter-Rater Agreement Training – Things to Consider: • Synchronous and Asynchronous • Experience of the Participants• How to Ensure Fidelity • Use of Video Examples• Use of Case Studies• Bias Exists• Single versus Multiple Observations• Professional Judgment• Degree of Familiarity
9
Getting Started in YOUR District• Getting Your Bearings• Developing a Guidebook• Developing a Training Protocol
10
I. Getting Your BearingsPilot with Administrative Team1. Pilot Approach: Start with Safe Example:
• Use a video from YouTube• Assess Specific Area of Teaching
• Student Engagement• Teacher Activity• Classroom Environment
2. Review current issue/goal/vision/needs3. Review current practices/instrumentation
• Rubrics• Protocol• Training
11
II. Developing a Guidebook• Paper and Electronic: Synchronous and Asynchronous• EXCELLENT Administrative Council Endeavor –
Ultimately to be used during Professional Learning Times with the Staffs
• Pieces and Parts• Terms/Definitions: Frame of Reference/Common Understanding
• “Demonstrates”• “Frequently”• “Usually”
12
II. Developing a Guidebook• Pieces and Parts (continued)
• Examples and Rubrics• Areas to Observe
• Establishes/maintains a culture of safety, respect and rapport
• Knows how to make content accessible to all learners
• Plans standards-based instruction and formative assessments
• Uses a variety of instructional strategies to meet needs
• Deepens understanding through critical thinking, problem-solving and student reflection
• Analyzes student performance and provides feedback
• Collaborates to support student learning
13
II. Developing a Guidebook• Pieces and Parts (continued)
• Exemplars: Video and Artifacts• Master-Coded Videos: Leads to high level of
agreement• Example Artifacts
• Classroom Rules• Lesson Plans• Parent Note Home
• Lesson Plan for Adults: Andragogy
14
III. Developing a Training Protocol• Step-by-Step Procedure1. Initial Training: Goals/Terms
• Practice Interpreting Rubrics• Differentiate between Different Performance Levels with Examples
2. Calibration of the Players• Video and Artifacts
3. Review of the System4. Adjustments (If needed)5. Implementation6. Evaluate/Reflect/Adjust7. Re-Calibrate…
15
• Start Slowly• Start with
“Engagement” with a Small Group
• Get Agreement• Gain Confidence
Over Time• Write and Refine in
Teams
Take Aways – To Do List
16
• Start with the End in Mind• Develop Common
Understandings• Use Groups to Ensure
Buy-In• Build Measurable Steps to
Reach Each Goal
Final Thoughts
17
Contact InformationTeacher Licensure and Accreditation Team DirectorScott [email protected]