Craft Effective Assessments with Illuminate and INSPECT… · Craft Effective Assessments with Illuminate and INSPECT John Endahl Ingham Intermediate School District ... • Ellen

Craft Effective Assessments with

Illuminate and INSPECT John Endahl

Ingham Intermediate School [email protected]

Session Overview

Planning the Assessment

Target-Instruction-Assessment Alignment

Test Blueprint

INSPECT and the Test Blueprint

Essential Questions

What is the purpose of the assessment?

Who will use the information?

How will the information be used?

Essential Questions Formative View

What is the purpose of the assessment?

To provide teachers with immediate information on student learning

Who will use the information?

Teachers and students in the classroom

How will the information be used?

To inform next steps in the learning process

Essential Questions Summative View

What is the purpose of the assessment?

Educator evaluation and accountability

Who will use the information?

Teachers and administrators

How will the information be used?

To certify the learning process

Effective Assessments

…are reliable - gives consistent results every time it is used.

…yield valid data - measures what it is supposed to measure.

The goal: Valid and reliable

The question: How do we achieve this?

Test Blueprint A test blueprint is the best way to ensure your assessment is

valid and reliable, if you follow it while developing the

assessment.

Sample Blueprint Item Type Depth of Knowledge Level Row Totals

S-R C-R C-R Perf (cell entries are points)

Brief Extended Recall Skill/Concept

Strategic Thinking

Extended Thinking Item Target

MA.5.OA.1 x 1 1

MA.5.OA.1 x 1 1

MA.5.OA.1 x 2 2 4

MA.5.MD.1 x 3 3

MA.5.MD.1 x 5 5 8

Column Totals 2 2 1 0 1 6 5 0 12 12

Item & Point Checksums 5 12 # of items 5

Key Elements

Learning Target/Item Alignment

Balance of Representation

Target-Method Match

High Quality Items

Learning Target/Item Alignment

Learning Targets

Identified by Type (Bloom’s Taxonomy, Webb’s DOK, etc.)

Appropriate, measurable, enduring

Measured with the most effective, appropriate method of assessment

Assessment method is consistent with the target, both in complexity and level of importance

Item Selection Are the items tightly aligned with the standards?

Do the assessment items address the targeted standards sufficiently at all levels of cognitive demand?

Are there an equal number of assessment items per standard? If not, is there a rationale for this?

Are there enough assessment items per standard to define mastery?

Balance of Representation

Items are balanced and should reflect what was taught and how it was taught within the classroom.

The assessment as a whole should reflect the emphasis of the lesson, unit or experience on which you are assessing.

Learning Target/Method Match

Identify the Depth of Knowledge (DOK) of the standard.

Match the cognitive demand of the standard with an appropriate item type.

Determine which item types will be included in the assessment and in what proportion.

Bloom & Webb Remember (Knowledge)

The recall of specifics and universals, involving little more than bringing to mind the appropriate material.

Recall Recall of a fact, information or procedure (e.g. What are three critical

skill cues for the overhand throw?)

Understand (Comprehension) Ability to process knowledge on a low level such that the knowledge can be reproduced or communicated without a verbatim repetition.

Apply (Application) The use of abstraction in concrete situations.

Skill/Concept Use of information, conceptual knowledge, procedures, two or more

steps, etc.

Analyze (Analysis) The breakdown of a situation into its component parts.

Strategic Thinking Requires reasoning, developing a plan or sequence of steps; has some

complexity; more than one possible answer.

Evaluate and Create (Synthesis and Evaluation)

Putting together elements and parts to form a whole, then making value judgements about the method.

Extended Thinking Requires an investigation; time to think and process multiple

conditions of the problem or task.

Webb’s Depth of Knowledge The verbs used in an item or

standard often relate directly to the Depth of Knowledge

Level 1 - Recall

Level 2 - Skill/Concept

Level 3 - Strategic Thinking

Level 4 - Extended Thinking

Activities

Percent of Common Core ELA Standards at each DOK level

Educational Policy Improvement Center. (2011). Lining Up: The Relationship between the Common Core State Standards and Five Sets of Comparison Standards. Eugene, Oregon

7%!

12%!

55%!

26%!

Level 3 - Strategic Thinking

Percent of Common Core Math Standards at each DOK level

Educational Policy Improvement Center. (2011). Lining Up: The Relationship between the Common Core State Standards and Five Sets of Comparison Standards. Eugene, Oregon

21%!

54%!

20%!

5%!

Level 2 - Skills/Concepts

Target-Method Match

Target to be Assessed

Assessment Method

Selected Response/Short-Response Extended Response Performance

AssessmentObservation/Conversations

Knowledge/Recall Good Match Good Match Not a good match Partial Match

Reasoning/Strategic Thinking Partial Match Good Match Good Match Good Match

Performance Skills Not a good match Not a good match Good Match Partial Match

Application/Products Not a good match Partial Match Good Match Not a good match

Methods of Assessment Selected Response Constructed

ResponsePerformance Assessment

Observations/Conversations

•  Multiple Choice•  True-False•  Matching

•  Diagram•  Fill in the blank

(words, phrases)•  Essay•  Short answer

(sentences, paragraphs)

•  Web•  Concept Map•  Flowchart•  Graph•  Table•  Matrix•  Illustration

•  Presentation•  Movement•  Science lab•  Athletic skill•  Dramatization•  Enactment•  Project•  Debate•  Model•  Exhibition•  Recital•  Performance task

•  Oral questioning•  Observation•  Interview•  Conference•  Process description•  Checklist•  Rating scale•  Journal sharing•  Describe a process•  Student self-

assessment•  Peer review

Test Blueprint Putting it all together

Standards

Specific Learning Targets derived from standards

Target DOK level

Assessment Method

Number of items/points

Sample Blueprint Item Type Depth of Knowledge Level Row Totals

S-R C-R C-R Perf (cell entries are points)

Brief Extended Recall Skill/Concept

Strategic Thinking

Extended Thinking Item Target

MA.5.OA.1 x 1 1

MA.5.OA.1 x 1 1

MA.5.OA.1 x 2 2 4

MA.5.MD.1 x 3 3

MA.5.MD.1 x 5 5 8

Column Totals 2 2 1 0 1 6 5 0 12 12

Item & Point Checksums 5 12 # of items 5

Selecting the standards

Selecting response type(s)

Filter on Depth of Knowledge

Looking at the item attributes

Assessment construction dashboard

Question Groups

Question Groups in Illuminate

Test Booklet in Illuminate

Adjusting point values

INSPECT and this process INSPECT’s search function makes it fairly easy to search for items to match your assessment blueprint.

The fact that all items are keyed to both DOK and Bloom’s is a huge advantage, but always verify the match accuracy.

The inability to assign point values to rubric-scored items is one of the biggest obstacles to using INSPECT with this assessment process.

Illuminate reports allow you to measure the effectiveness of the assessment as well as the effectiveness of your instruction.

Keys to Quality Classroom

Assessment

Michigan Assessment Consortium

http://www.michiganassessmentconsortium.org/common-assessment-module-series

Special Thanks

•  Michigan Assessment Consortium

•  Ellen Vorenkamp, Wayne RESA

•  Kathryn Dewbury-White, Michigan Assessment Consortium

Thank you!

John Endahl

Ingham Intermediate School District

[email protected]