Welcome! Making Formative and Benchmark Assessments Count WERA Conference December 2007 1.

Welcome!

Making Formative and Benchmark Assessments Count

WERA Conference

December 2007

1

Goals for the Afternoon

Share lessons learned from the ASK Project (an assessment design project funded by the National Science Foundation). Designing assessment systems for science curriculum.

Understanding best practices.

2

Purposes

Classroom assessment practicesDevelop assessment strategies and practices that lead to better student achievement and enhance instructional practices.

AccountabilityDevelop assessments with the technical quality needed to provide accountability information to districts.

3

External Evaluator

BEAR

Charleston

ME ST LP LF

Fresno

ME ID SE MX

Las Vegas

MS IDLF SE EV

Manhattan

HB ST LF LP

Vista

MS FN

Vancouver

PS HB FN MD

San Diego

PS EV

Formative Evaluator

Advisory Board

Madison

WA EM VB MX

Phoenix

WA EM VB MD

LHS

4

Two Reports from NRC

5

The Assessment Triangle

OBSERVATION INTERPRETATION

COGNITION

6

The Assessment Triangle

OBSERVATION INTERPRETATION

COGNITION

7

Cognition

The Cognitive Perspective Focuses on how people develop

structures of knowledge—content, reasoning, and problem solving.

The Situative Perspective (Sociocultural) Focuses on how people learn to

participate in the practices, goals, and habits of mind of a particular community.

8

Inquiry

Constructs

9

Physics of Sound

Constructs

10

Progress Levels

Strategic

Conceptual

Recognition

Notions

11

The Assessment Triangle

OBSERVATION INTERPRETATION

COGNITION

12

Observation—Item Development

Because you can’t see directly into a child’s mind, you have to find ways to infer evidence of learning. Items are written Panels review items Researchers look at cognitive demands Students and teachers test them out! Empirical data informs further revisions

13

ASK Assessment Items

Embedded Assessment items (diagnostic and part of daily instruction) Notebook entries Notebook sheets Response sheets

Benchmark Assessment items (evaluative, but also formative—given periodically)

Short Answer Open Response Multiple Choice/Multiple Answer

14

Benchmark Assessments

Before starting After Investigation 1 After Investigation 2 After Investigation 3 After Investigation 4 After completing the

module

Survey Inv 1 I-Check Inv 2 I-Check Inv 3 I-Check Inv 4 I-check Posttest

15

The Assessment Triangle

OBSERVATION INTERPRETATION

COGNITION

16

Coding (scoring) guides—

Based on “Progress Levels”

4 Strategic

3 Conceptual

2 Recognition

1 Notions

0 Makes no attempt

(Coding guides are 0-2, 0-3 or 0-4 depending on the level of the question.)

17

Qualitative Methods Teacher feedback, expert panels and

cognitive analysis.

Quantitative Methods Item Response Theory (statistical analysis) ClassMap (computer program/reports)

Interpretation

18

The Assessment Triangle

OBSERVATION

(Item development

)

INTERPRETATION

(Coding guides and measurement

model)

COGNITION

(Construct Maps &

Progress Maps) 19

Focus on Embedded Assessment

How is embedded assessment enacted within a curriculum?

Context: Physics of Sound

20

What students have learned before the lesson we’ll be doing

Sound is caused by vibrations. A sound source is an object that is vibrating.

A sound receiver detects sound vibrations.

Sounds have identifiable properties.

Pitch (a property of sound) is how high or low the sound is (volume = loud or soft).

The faster an object vibrates the higher the pitch (and vice versa).21

Notes to Myself

Review the At-a-Glance chart Read Background for the Teacher and

Teaching Children About… Read through the steps in Guiding the

Investigation

Review the I-Check for Investigation 2

Review the focus questions

Prep for Inv 2

22

Notes to Myself

The three big ideas:

Sounds are caused by vibrations Clarify distinction between pitch and volume

The length of the instrument affects the pitch

Shorter length = higher pitch Longer length = lower pitch

Tension also affects the pitch More tension = higher pitch Less tension = lower pitch

Prep for Inv 2

23

Focus Question

How does length affect the pitch (and therefore the speed of vibrations)?

Record observations:WaterphoneXylophoneKalimbaString Beam

24

Class Discussion

How does length affect the pitch (and therefore the speed of vibrations)?

Record observations:WaterphoneXylophoneKalimbaString Beam

25

Notes to MyselfLength/Pitch

waterphonena √ √ √ √ √ √ na

xylophone √ rev ? ? rev ? ? √

kalimba √ √ rev P/v ? da ? na

String beam √ √ ? √ rev ? ? na

Ada

Badu

Cala

Derek

Edita

Flavo

Gen

Halyn

Key

√ = Got it!

rev = reversed

? = confused

da = didn’t answer the question

na = no answer

26

Notes to MyselfLength/Pitch

27

Can students distinguish high from low pitches?

Do they know the rule?

Why are they inconsistent?

Next Steps

(Can students discriminate pitch?)

Activity:Close your eyes.Listen to the sounds.If the sound has a high pitch, show thumbs up.If the sound has a low pitch, show thumbs down.

28

What are our “rules?”

The longer the length,

the ___________ the pitch,

29

and the ___________ the vibrations.

What are our “rules?”

The shorter the length,

the ___________ the pitch,

30

and the ___________ the vibrations.

Next Steps

Statement disagrees with the rule

Statement needs clarification

Statement agrees with the rule

31

•The longer the length, the lower the pitch, the slower the vibrations.

•The shorter the length, the higher the pitch, the faster the vibrations.

32

33

Next Steps

Statement disagrees with the rule

Statement needs clarification

Statement agrees with the rule

34

•The longer the length, the lower the pitch, the faster the vibrations.

•The shorter the length, the higher the pitch, the slower the vibrations.

35

Next Steps

What do the dots tell you about your use of the “rule?”

Rewrite any sentences you marked with a red or yellow dot.

36

37

Purposes:Summative (measure for giving grades)Formative (continue the learning)

Procedures:Take the testCode, but don’t mark on papersUse self-assessment strategies with students before revealing codes.

38

Benchmark Assessments

Students have Learned… Sound is caused by vibrating objects. Length affects pitch

Longer = lower pitch (slower vibrations) Shorter = higher pitch (faster vibrations)

Tension affects pitch Looser = lower pitch (slower vibrations) Tighter = higher pitch (faster vibrations)

39

Benchmark Assessment Take the I-Check for Investigation 2

40

Behind the scenes

After students take the benchmark, the teacher codes the items to determine what students know and what they still need help with.

The teacher then plans which items to use to help students self-assess their understanding.(we’ll come back to coding later) 41

Self-assessment Strategy Multiple-Choice Corners

Look at item 13

• You will be assigned a letter A, B, or C• (In the classroom students would go to the corner that corresponds with their actual answer.)

• Meet with your group to create an argument to convince the other groups in the room that your answer is the correct one.

• If at some point you “disagree with yourself,” you are allowed to change corners. 42

Self-assessment Strategy Sentence Starters

I used to think….but now I think… I should have gotten this one right, I just…

I know…but I’m still not sure about… The most important thing to remember is…

Can you help me with… Next time I will remember to… I’m confused about… Now I know…

43

Sentence StartersAndy

44

Sentence StartersAndy

45

Sentence StartersRachael

46

Sentence StartersRachael

47

Coding Benchmark ItemsCode If the student…

3

states that the short string will have a higher pitch than the long string (does not need to use the word pitch, but must describe the sound of both strings or make a correct comparison).

2

states that the short string will have a high pitch or the long string will have a low pitch, but does not make a comparison.

1 provides any other answer.

0 makes no attempt.Note: Coding Guides may be 0–2, 0–3 or 0–448

44

Levels of Progress

4 Strategic

3 Conceptual

2 Recognition

1 Notions

0 No attempt

49

Code Inv 2 I-Check

Review the coding guide for first item.

Code responses for all students on that item

Go on to the next item

22 23 24 25 26a

Ada 3

Badu 2

Cala 1

Derek 2

50

51

The Development Process Construct Maps and Frameworks Item Development Field Testing Analysis using item response theory software

(ConstructMap) Item fit Standard setting

Iterative revision of items Progress Levels and Maps

52

Does formative assessment matter?

Benchmark scores were not as high as we thought they should be.

When we interviewed teachers they said they looked over students’ shoulders when they wrote in notebooks, but did not critically look at embedded assessment evidence.

Through FAST Project, worked with a small group of teachers to “force” looking at embedded assessments. 53

1.6 1.2 2.61.615.7 15.9

9.7

22.9

34.9

87.1

60.0

46.7

0%

20%

40%

60%

80%

100%

1 2 3

Series4Series3Series2Series1

Strategic

Conceptual

Recognition

Notions

54

Magnetism and Electricity

Posttest Results

ClassMap

Berkeley Evaluation and Assessment Research Center

55

Beliefs and Actions

The most valuable use of assessment is as a learning

tool.

• Embedded assessment is the major focus.

• Assessment data is used to identify gaps in content understanding for both students and teachers.

56

Beliefs and Actions

Students need to take

responsibility for their own

learning through active investigation and reflection.

• Collaboration rather than competition.

• Students have time to reflect on their learning through class interaction and writing-to-learn activities.

• Students are willing to make their thinking public…

• Students self-assess rather than self-evaluate… 57

Beliefs and Actions

Embedded assessment needs to

reflect student thinking and how it changes and develops.

• Students write initial thinking in their notebooks without teacher intervention.

• Students may add additional information (under line of learning) after feedback, class discussion, or further activities.

58

Beliefs and Actions

Students need immediate and effective feedback to improve learning.

• Teachers work to have thorough understanding of the content…

• Teachers look at notebooks after every lesson to reflect on learning and instruction and to determine next-steps.

• Teachers provide feedback as soon and as often as possible.

• Feedback is based on data and provides information for improved understanding.

59

Discussion

Questions?

Comments?

60