From Evidence to Action: A Seamless Process in Formative Assessment? Margaret Heritage Jinok Kim Terry Vendlinski American Educational Research Association.

From Evidence to Action: A Seamless Process in Formative Assessment?

Margaret Heritage

Jinok Kim

Terry Vendlinski

American Educational Research AssociationAnnual Meeting

New York, NY - March 24-28, 2008

Evidence to Action: A Seamless Process?

• A Generalizability study of measures of teacher knowledge

• Teachers can draw appropriate inferences from evidence

• Difficulties in deciding next instructional steps

Formative Assessment

Formative Assessment

• An ongoing process to gather evidence and provide feedback about learning

• Use the feedback to identify the gap between current learning and desired goals (Sadler, 1989)

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Teacher Knowledge Measures

Measures of Teacher Knowledge for Teaching Mathematics

• Drawn from POWERSOURCE

• Conceptualize teacher knowledge as central to and embedded in the everyday practice of teaching

• Key principles underlying mastery of algebra I

Distributive property

Rational number equivalence

Solving equations

Student Assessment

Student Assessment

Student Assessment

Student Assessment

Teacher Knowledge Measures

1. What is the key principle that these assessments address? Why do students need to understand this principle for algebra I?

2. What inferences would you draw from this student’s responses? What does this student know? What does this student not know?

3. If this student were in your class, based on your responses to questions 2 and 3, what would you do next in your instruction?

4. If you were this child’s teacher, what written feedback would you give to this student?

Scoring Rubric: Next Instructional Steps for Teaching the Distributive PropertyScore Criteria

4 Explain the distributive property as repeated additionExplain factoring as distribution in reverseModel the use of the distributive property with whole numbersModel generalizing to other numbers and variables

3 EitherExplain the distributive property as repeated additionOrExplain factoring as distribution in reverseModel the use of the distributive property with at least whole number

2 Explain procedures for how to use the distributive property, equating procedures with the order of operations

1 No explanation of the distributive property

Generalizability (G) Study

G-Theory Framework

• Object of Measurement:

Teachers’ pedagogical knowledge in mathematics

• Aims:

To assess the generalizability of a task score in a single condition to an average score in a countless combination of conditions

Determine the source of error if the task score is not generalizable

Initial Assumptions: 3 Potential Sources of Measurement Error

• Variations in raters’ scoring of a teacher’s responses

• A teacher’s knowledge of principles may vary

• A teacher’s score may vary depending on the task

Sample of Participating Teachers

• 118 6th grade teachers from a variety of districts in Los Angeles County

• 90% credentialed for K-6, 60% for 7-8, 34% grade 9, 20% for 10-12

• 90% general credential

• 14% mathematics credential

Analysis

o x r x p x t

object of measurement

(teacher)

rater principle type of task

Variability Findings: Main Effects

• Main effect of raters = 0.2%

The impact of raters is negligible

• Main effect of type of task = 25%

Teachers’ scores may not be generalizable across tasks

Variability Findings: Interactions

• Interactions of object of measurement (teacher), principle, and task = 39.4%

Some principle-task combinations are more difficult for some teachers than for other teachers

• Interactions of principle and task = 8.2%

Some combinations of principle and task more difficult

Variable N Mean Std Dev Min Max

Principle: distributive property

Task: identifying key principle 114 2.07 0.63 1 3.67

Task: evaluating student understanding

113 2.14 0.94 1 4.00

Task: planning next instruction 113 1.21 0.36 1 2.00

Principle: solving equations

Task: identifying key principle 112 1.82 0.89 1 3.83

Task: evaluating student understanding

111 2.06 0.93 1 3.83

Task: planning next instruction 112 1.21 0.39 1 2.83

Principle: rational number equivalence

Task: identifying key principle105 2.94 0.47 1 4.00

Task: evaluating student understanding

102 2.07 0.98 1 4.00

Task: planning next instruction 101 1.36 0.48 1 2.83

Improving the Translation of Evidence to Action

Know Which Way to Go?

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Learning from Teaching

And Finally …

• Challenge of developing valid and reliable measures of teacher knowledge

• Using assessment information to plan subsequent instruction tends to be the most difficult task

• Evidence may provide the basis for action but cannot itself “form” the action

[email protected]