1 Using K-12 Assessment Data from Teacher Work Samples as Credible Evidence of a Teacher Candidate’s Ability to Produce Student Learning Presented by Roger.

1

Using K-12 Assessment Data from Using K-12 Assessment Data from Teacher Work Samples as Credible Teacher Work Samples as Credible Evidence of a Teacher Candidate’s Evidence of a Teacher Candidate’s

Ability to Produce Student LearningAbility to Produce Student LearningPresented by

Roger Pankratz

Tony NormanJudy Pierce

of

Western Kentucky UniversityPresented at

The Making an Impact Conference

Kennesaw University March 2007

2

Our Leading Question

Can P-12 assessment data on teacher work samples produced by student teachers provide credible evidence that Western’s graduates have the ability to impact learning with the students they teach?

3

Our Hypothesis

Work sample data can provide

credible evidence of ability to

impact learning provided unit

learning objectives and

assessments to measure

learning meet credibility

standards.

4

Our Research Questions

1) What are the gains in K-6 student achievement reported by Western’s student teachers on teacher work samples?

2) What is the quality of assessments and degree of alignment between unit objectives and assessments used in student teachers’ work samples at Western?

3) To what extent do unit objectives in Western’s student teacher work samples address Kentucky’s core content standards?

5

Our Rationale for Exploring the Use of Work Sample Data as a

Measure of Impact

Renaissance Partnership work (Denner et al., 2004) demonstrating relationship between overall TWS performance and quality of learning assessments

Institutional work (e.g., Idaho State and Longwood) suggesting that P-12 scores within the TWS could be used to ascertain impact on learning

6

What is a Teacher Work Sample?

Provides documentation of a teacher candidate’s ability to plan, deliver, and assess a complete standards-based unit of instruction

Provides instruction embedded evidence of the impact of teaching on student learning

7

Validity & Reliability Evidence

Processes parallel targeted INTASC and Kentucky Teacher Standards

Experts affirm that processes describe behaviors/skills that are frequent, critical, and necessary for good teaching.

Multi-institutional studies (Denner et al., 2004) show that TWS can be scored reliably

Two WKU studies show between 81-90% inter-rater agreement. An additional study showed between 67%-95% agreement (faculty tended to score higher) but some training discrepancies were present.

8

The Context of Teacher Work Sample Use at Western Kentucky

University

Western Kentucky University has about 400 student teachers each year:

300 K-5

100 6-12

Western has required TWS as a performance assessment in student teaching since 2001.

9

The Context of Teacher Work Sample Use at Western Kentucky

University

Student teachers upload electronic versions of their teacher work samples on the College of Education’s data management system.

10

The Context of Teacher Work Sample Use at Western Kentucky

University

Western convenes teacher educators and school practitioners each year to independently score TWS’s (semi-holistic and holistic).

University student teacher supervisors initially score TWS’s for student teaching evaluation.

Performance data of TWS’s are used for program improvement.

11

The Seven Teaching Processes Assessed in

Western’s Teacher Work Sample

1) Use of teaching context

2) Unit learning objectives

3) Assessment plan

4) Instructional design

5) Instructional decision making

6) Analysis of learning results

7) Reflection on teaching and learning

12

The Design of our Initial Study of Teacher Work Sample Assessment Data, Quality of Assessments and

Quality of Unit Objectives

Analysis of reported pre-post assessment student gains in 22 student teacher work samples produced in fall of 2006 (460 students)

Consensus rating of the quality of assessments developed and administered by each student teacher

13

The Design of our Initial Study of Teacher Work Sample Assessment Data, Quality of Assessments and

Quality of Unit Objectives Description of assessments used for

each unit learning objective number and type (short answer, multiple choice, true/false, open response, matching)

“Quick and dirty” rating of the quality of instructional objectivesAlignment with Kentucky’s content

standardsImportance of objective related to

content standards

14

Study Results:Impact on Student Learning

0.44

0.84

00.20.40.60.8

1

Pre Post

Goal 1

0.27

0.77

00.20.40.60.8

1

Pre Post

Goal 2

0.42

0.82

00.20.40.60.8

1

Pre Post

Overall

Significant Student Gain (Controlling for Student Teacher Effects)

Student Teacher N = 22, P-12 Student N = 464

15

Study Results: Quality of Assessments Rubric

16

Study Results: Quality of Assessments

All TWS had two levels of DOK with most having a lower level goal (DOK 1,2) and higher level goal (DOK 3,4).

Total QA Scores ranged from 9 to 15 (M = 11.89, SD = 1.73).

Holistic QA Scores Frequencies: “2” = 14 (64%), “3” = 3 (27%), and “4” = 2 (9%)

No significant relationships between Faculty Holistic TWS Score and either Total QA Scores (Pearson r = -.25) or Holistic QA Scores (r = -.18)

No significant relationships between Total Gain Scores and either Total QA Scores (Pearson r = .34) or Holistic QA Scores (r = .31)

17

Study Results: Description of Assessments for 10 Work Samples Selected at Random

1 – Kindergarten 3 – Science

3 – Grade 1 4 – Math

2 – Grade 3 2 – Health

2 – Grade 41 – Social Studies

2 – Grade 6

18

Study Results: Description of Assessments

Average number of assessment items per unit of instruction – 15.515.5

Range of assessment items for a unit of instruction: 8 – 308 – 30

Average number of assessment items per instructional objective: 55

Range of assessment items for an instructional objective: 1 - 131 - 13

19

Study Results: Description of Assessments

Types of Assessments Used Overall Short answer: 5353

Multiple choice: 48

Matching: 2424

True/False: 1717

Open Response: 1313

20

Study Results: Description of Assessments

Types of assessments used by unit objective Depth of Knowledge (DOK)

DOK1 DOK2 DOK3 DOK4

Short Answer 15 25 5 8

Multiple Choice 14 14 20 0

Matching 6 13 5 0

True/False 7 5 1 4

Open Response 3 5 4 1

21

Study Results: Quality of Objectives

Alignment with Kentucky Core Content

4 – 0

3 – 4

2 – 3

1 – 2

Importance of objective relative to standards

4 – 1

3 – 7

2 – 1

1 – 0

22

CONCLUSIONS

1) TWS assessment data have potential for credibility provided standards for instructional objectives and assessments are addressed.

2) TWS show impressive average student gains but there is a large variability by individual student.

23

CONCLUSIONS

3) The quality of assessments used looks promising but needs improvement.

4) The number of assessment items per objective for the ten work samples reviewed is too small for any degree of reliability.

24

CONCLUSIONS

5) The quality of instructional objectives for the ten work samples reviewed varies widely with a majority of objectives needing improvement in alignment and focus.

6) More instruction is needed on matching assessments to objectives – especially assessments for DOK 3 and 4 level objectives.

25

CONCLUSIONS

7) With a sustained effort to provide instruction for teacher candidates in the development of high quality instructional objectives and assignments, data from work samples could be one source of evidence of teacher graduate’s potential for impact on student learning.

26

Next Steps

Conducting larger, more precise TWS studies to explore, such as…

The extent to which all TWS scores can be accessed and matched to goalsThe quality of the assessments (depth of knowledge, alignment to standards, adaptations to meet diverse needs of students)The potential for differential impact based on student-teacher program and P-12 student characteristics

Working with state to connect TWS, as well as other program assessments, success to impact on P-12 student learning during teaching internship

27

Discussion Question 1

What are your reactions to and/or comments on Western’s exploration of teacher work sample data as one source of impact evidence?

28

Discussion Question 2

What suggestions do you have for next steps Western might consider in it’s effort to develop credibility in student teacher work sample data?

29

Discussion Question 3

What would you suggest as standards for assessments and learning objectives to improve credibility of work sample data?

30

Discussion Question 4

What actions should Western take to engage other institutions in development of work samples that produce credible data?