Measuring Teaching with Teachers Lessons from the Measures of Effective Teaching Project ECET2 Florida May 3, 2013
Measuring Teaching with Teachers
Lessons from the Measures of Effective Teaching Project
ECET2 Florida May 3, 2013
2
Why do we focus on teaching?
Why start with measurement?
Multiple measures of effectiveness
Useful and actionable feedback
Targeted support & PD
Meaningful tenure
Differentiated roles
Strategic placement of
teachers
Elevated teaching profession
Improved student outcomes
3
Data Collection
•Two school years: 2009-10 and 2010-11
•>100,000 students
•Grades 4-8: ELA and Math
•High School: ELA I, Algebra I and Biology 4
MET Project Study Design
5
6
Key lesson from the MET project results:
Observers can provide reliable, objective feedback on
specific elements of instruction.
Caveats:
• Clear expectations based on a validated rubric.
• Raters should be certified, demonstrating accuracy
before beginning observations.
• Multiple observations.
Ensuring Reliable & Trustworthy
Observations
• Calibrate • Refine
• Test • Train
Video examples for anchor points
Rater certification: Don’t pass, don’t rate
Periodic tuning: Out of
tune, don’t rate
Adjustments to observation
framework based on data
7
Actual scores
for 7500
lessons.
Framework for Teaching (Danielson)
Un
sa
tis
facto
ry
Yes/no questions; posed
in rapid succession;
teacher asks all
questions; same few
students participate.
Ba
sic
Some questions ask for
explanations; uneven
attempts to engage all
students.
Pro
fic
ien
t Most questions ask for
explanation; discussion
develops, teacher steps
aside; all students
participate.
Ad
va
nc
ed
All questions high quality;
students initiate some
questions; students
engage other students.
© 2013 Bill & Melinda Gates Foundation | 8
Many roads to reliability
One more
lesson +.07
One more observer
+.16
© 2013 Bill & Melinda Gates Foundation | 9
10
Key lesson from the MET project results:
Students can provide feedback on specific aspects
of their classroom experience, which is both reliable
and predictive of student achievement.
Caveats:
• Questions should focus on specific aspects of practice
(not a popularity contest).
• Students should fill out surveys confidentially to lessen
pressure from peers and others.
Tripod Student Perception survey
Survey Statement Category Rank
1
2
3
5
Students in this class treat the teacher with respect
My classmates behave the way my teacher wants them to
Control
Control
Control Our class stays busy and doesn’t waste time
Challenge
• In this class, we learn a lot every day Challenge In this class, we learn to correct our mistakes
33 I have learned a lot this year about [the state test] Test Prep
34 Getting ready for [the state test] takes a lot of time in our class Test Prep
4 In this class, we learn a lot every day
11
12
Key lesson from the MET project results: Any
single measure is open to manipulation. • If test scores alone, risk “teaching the test.”
• If student surveys alone, risk pandering.
• If observations alone, teachers using unconventional
teaching methods would be forced to conform.
13
14
15
How much does practice differ?
16
Role of teachers in the MET project
What do you think?
What should the foundation do to ensure findings from the MET project are used to advance high-quality teacher evaluation?
How should the foundation engage teachers
in future research projects?
17
Thank You
© 2013 Bill & Melinda Gates Foundation. All Rights Reserved. Bill & Melinda Gates Foundation is a registered trademark in the United States and other countries.
@szbuhayar
Improvement-focused evaluation systems
19
Monitor validity
Ensure reliability
Assure accuracy
Make meaningful distinctions
Prioritize support and feedback
Use data for decisions at all levels
Set expectations
Use multiple measures
Balance weights
Closing the effectiveness gap
20
21
Preliminary Results
Multiple Measures are more strongly related to student performance gains that Master’s Degrees or Experience
22
-.1
-.0
5
0
.05
.1
Actu
al A
ch
ievem
ent a
fter
Ra
nd
om
Assig
nm
ent
-.1 -.05 0 .05 .1Predicted achievement using teacher's past measures of teaching.
Note: Teachers were sorted into 20 groups by their predicted student achievement relative to the randomization group mean.
Means are reported for each of the 20. Predictions are adjusted for non-compliance.
of Randomized Classrooms (Math)
Figure 1. Actual and Predicted Achievement
Actual = Predicted
Final Results
23
Students with Most Effective Teachers Learn More in School
Do students of higher-scoring teachers learn more?
24
25
1
2
3
4
5
6
7
0 0.25 0.5 0.75 1
CLASS Score Distribution
26
1
2
3
4
5
6
7
0 0.25 0.5 0.75 1
CLASS Score Distribution
27
1
2
3
4
5
6
7
0 0.25 0.5 0.75 1
CLASS Score Distribution
0.68 points
28 Invest in Improvement
Prioritize support & feedback
29
Domain Expectation Indicator
Providing
Feedback
on
Strengths
and Areas
of Growth
Communicating
Strengths S.1
The teacher’s strength is exposed in a way that encourages
reflection on how that strength impacts student learning.
Communicating
Area of Growth
G.1
The teacher’s area of growth and related evidence from the
lesson is exposed in a way that encourages reflection on how
that area impacts learning and specific next steps.
Invest in Improvement
Use data for decisions at all levels