Welcome! Making Formative and Benchmark Assessments Count WERA Conference December 2007 1
Welcome!
Making Formative and Benchmark Assessments Count
WERA Conference
December 2007
1
Goals for the Afternoon
Share lessons learned from the ASK Project (an assessment design project funded by the National Science Foundation). Designing assessment systems for science curriculum.
Understanding best practices.
2
Purposes
Classroom assessment practicesDevelop assessment strategies and practices that lead to better student achievement and enhance instructional practices.
AccountabilityDevelop assessments with the technical quality needed to provide accountability information to districts.
3
External Evaluator
BEAR
Charleston
ME ST LP LF
Fresno
ME ID SE MX
Las Vegas
MS IDLF SE EV
Manhattan
HB ST LF LP
Vista
MS FN
Vancouver
PS HB FN MD
San Diego
PS EV
Formative Evaluator
Advisory Board
Madison
WA EM VB MX
Phoenix
WA EM VB MD
LHS
4
Two Reports from NRC
5
The Assessment Triangle
OBSERVATION INTERPRETATION
COGNITION
6
The Assessment Triangle
OBSERVATION INTERPRETATION
COGNITION
7
Cognition
The Cognitive Perspective Focuses on how people develop
structures of knowledge—content, reasoning, and problem solving.
The Situative Perspective (Sociocultural) Focuses on how people learn to
participate in the practices, goals, and habits of mind of a particular community.
8
Inquiry
Constructs
9
Physics of Sound
Constructs
10
Progress Levels
Strategic
Conceptual
Recognition
Notions
11
The Assessment Triangle
OBSERVATION INTERPRETATION
COGNITION
12
Observation—Item Development
Because you can’t see directly into a child’s mind, you have to find ways to infer evidence of learning. Items are written Panels review items Researchers look at cognitive demands Students and teachers test them out! Empirical data informs further revisions
13
ASK Assessment Items
Embedded Assessment items (diagnostic and part of daily instruction) Notebook entries Notebook sheets Response sheets
Benchmark Assessment items (evaluative, but also formative—given periodically)
Short Answer Open Response Multiple Choice/Multiple Answer
14
Benchmark Assessments
Before starting After Investigation 1 After Investigation 2 After Investigation 3 After Investigation 4 After completing the
module
Survey Inv 1 I-Check Inv 2 I-Check Inv 3 I-Check Inv 4 I-check Posttest
15
The Assessment Triangle
OBSERVATION INTERPRETATION
COGNITION
16
Coding (scoring) guides—
Based on “Progress Levels”
4 Strategic
3 Conceptual
2 Recognition
1 Notions
0 Makes no attempt
(Coding guides are 0-2, 0-3 or 0-4 depending on the level of the question.)
17
Qualitative Methods Teacher feedback, expert panels and
cognitive analysis.
Quantitative Methods Item Response Theory (statistical analysis) ClassMap (computer program/reports)
Interpretation
18
The Assessment Triangle
OBSERVATION
(Item development
)
INTERPRETATION
(Coding guides and measurement
model)
COGNITION
(Construct Maps &
Progress Maps) 19
Focus on Embedded Assessment
How is embedded assessment enacted within a curriculum?
Context: Physics of Sound
20
What students have learned before the lesson we’ll be doing
Sound is caused by vibrations. A sound source is an object that is vibrating.
A sound receiver detects sound vibrations.
Sounds have identifiable properties.
Pitch (a property of sound) is how high or low the sound is (volume = loud or soft).
The faster an object vibrates the higher the pitch (and vice versa).21
Notes to Myself
Review the At-a-Glance chart Read Background for the Teacher and
Teaching Children About… Read through the steps in Guiding the
Investigation
Review the I-Check for Investigation 2
Review the focus questions
Prep for Inv 2
22
Notes to Myself
The three big ideas:
Sounds are caused by vibrations Clarify distinction between pitch and volume
The length of the instrument affects the pitch
Shorter length = higher pitch Longer length = lower pitch
Tension also affects the pitch More tension = higher pitch Less tension = lower pitch
Prep for Inv 2
23
Focus Question
How does length affect the pitch (and therefore the speed of vibrations)?
Record observations:WaterphoneXylophoneKalimbaString Beam
24
Class Discussion
How does length affect the pitch (and therefore the speed of vibrations)?
Record observations:WaterphoneXylophoneKalimbaString Beam
25
Notes to MyselfLength/Pitch
waterphonena √ √ √ √ √ √ na
xylophone √ rev ? ? rev ? ? √
kalimba √ √ rev P/v ? da ? na
String beam √ √ ? √ rev ? ? na
Ada
Badu
Cala
Derek
Edita
Flavo
Gen
Halyn
Key
√ = Got it!
rev = reversed
? = confused
da = didn’t answer the question
na = no answer
26
Notes to MyselfLength/Pitch
27
Can students distinguish high from low pitches?
Do they know the rule?
Why are they inconsistent?
Next Steps
(Can students discriminate pitch?)
Activity:Close your eyes.Listen to the sounds.If the sound has a high pitch, show thumbs up.If the sound has a low pitch, show thumbs down.
28
What are our “rules?”
The longer the length,
the ___________ the pitch,
29
and the ___________ the vibrations.
What are our “rules?”
The shorter the length,
the ___________ the pitch,
30
and the ___________ the vibrations.
Next Steps
Statement disagrees with the rule
Statement needs clarification
Statement agrees with the rule
31
•The longer the length, the lower the pitch, the slower the vibrations.
•The shorter the length, the higher the pitch, the faster the vibrations.
32
33
Next Steps
Statement disagrees with the rule
Statement needs clarification
Statement agrees with the rule
34
•The longer the length, the lower the pitch, the faster the vibrations.
•The shorter the length, the higher the pitch, the slower the vibrations.
35
Next Steps
What do the dots tell you about your use of the “rule?”
Rewrite any sentences you marked with a red or yellow dot.
36
37
Purposes:Summative (measure for giving grades)Formative (continue the learning)
Procedures:Take the testCode, but don’t mark on papersUse self-assessment strategies with students before revealing codes.
38
Benchmark Assessments
Students have Learned… Sound is caused by vibrating objects. Length affects pitch
Longer = lower pitch (slower vibrations) Shorter = higher pitch (faster vibrations)
Tension affects pitch Looser = lower pitch (slower vibrations) Tighter = higher pitch (faster vibrations)
39
Benchmark Assessment Take the I-Check for Investigation 2
40
Behind the scenes
After students take the benchmark, the teacher codes the items to determine what students know and what they still need help with.
The teacher then plans which items to use to help students self-assess their understanding.(we’ll come back to coding later) 41
Self-assessment Strategy Multiple-Choice Corners
Look at item 13
• You will be assigned a letter A, B, or C• (In the classroom students would go to the corner that corresponds with their actual answer.)
• Meet with your group to create an argument to convince the other groups in the room that your answer is the correct one.
• If at some point you “disagree with yourself,” you are allowed to change corners. 42
Self-assessment Strategy Sentence Starters
I used to think….but now I think… I should have gotten this one right, I just…
I know…but I’m still not sure about… The most important thing to remember is…
Can you help me with… Next time I will remember to… I’m confused about… Now I know…
43
Sentence StartersAndy
44
Sentence StartersAndy
45
Sentence StartersRachael
46
Sentence StartersRachael
47
Coding Benchmark ItemsCode If the student…
3
states that the short string will have a higher pitch than the long string (does not need to use the word pitch, but must describe the sound of both strings or make a correct comparison).
2
states that the short string will have a high pitch or the long string will have a low pitch, but does not make a comparison.
1 provides any other answer.
0 makes no attempt.Note: Coding Guides may be 0–2, 0–3 or 0–448
44
Levels of Progress
4 Strategic
3 Conceptual
2 Recognition
1 Notions
0 No attempt
49
Code Inv 2 I-Check
Review the coding guide for first item.
Code responses for all students on that item
Go on to the next item
22 23 24 25 26a
Ada 3
Badu 2
Cala 1
Derek 2
50
51
The Development Process Construct Maps and Frameworks Item Development Field Testing Analysis using item response theory software
(ConstructMap) Item fit Standard setting
Iterative revision of items Progress Levels and Maps
52
Does formative assessment matter?
Benchmark scores were not as high as we thought they should be.
When we interviewed teachers they said they looked over students’ shoulders when they wrote in notebooks, but did not critically look at embedded assessment evidence.
Through FAST Project, worked with a small group of teachers to “force” looking at embedded assessments. 53
1.6 1.2 2.61.615.7 15.9
9.7
22.9
34.9
87.1
60.0
46.7
0%
20%
40%
60%
80%
100%
1 2 3
Series4Series3Series2Series1
Strategic
Conceptual
Recognition
Notions
54
Magnetism and Electricity
Posttest Results
ClassMap
Berkeley Evaluation and Assessment Research Center
55
Beliefs and Actions
The most valuable use of assessment is as a learning
tool.
• Embedded assessment is the major focus.
• Assessment data is used to identify gaps in content understanding for both students and teachers.
56
Beliefs and Actions
Students need to take
responsibility for their own
learning through active investigation and reflection.
• Collaboration rather than competition.
• Students have time to reflect on their learning through class interaction and writing-to-learn activities.
• Students are willing to make their thinking public…
• Students self-assess rather than self-evaluate… 57
Beliefs and Actions
Embedded assessment needs to
reflect student thinking and how it changes and develops.
• Students write initial thinking in their notebooks without teacher intervention.
• Students may add additional information (under line of learning) after feedback, class discussion, or further activities.
58
Beliefs and Actions
Students need immediate and effective feedback to improve learning.
• Teachers work to have thorough understanding of the content…
• Teachers look at notebooks after every lesson to reflect on learning and instruction and to determine next-steps.
• Teachers provide feedback as soon and as often as possible.
• Feedback is based on data and provides information for improved understanding.
59
Discussion
Questions?
Comments?
60