ECD in the Scenario-Based GED ® Science Test Kevin McCarthy Dennis Fulkerson Science Content Specialists CCSSO NCSA June 29, 2012 Minneapolis This material.

ECD in the Scenario-Based GED® Science Test

Kevin McCarthyDennis FulkersonScience Content Specialists

CCSSO NCSAJune 29, 2012Minneapolis

This material is based on work supported by the National Science Foundation under grant DRL-0733172 (Application of Evidence-Centered Design in Large-Scale Assessment). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

NSF-Funded Research & GED® Science

Study the incorporation of evidence-centered test design (ECD) in technology-based large-scale assessment

Goal: Determine how ECD can enhance the quality of technology-based large scale assessment and make development more efficient

Partners:

2 ECD in GED® Science

Incorporating ECD into GED® Science

PurposeTo develop valid scenarios and items that allow us to evaluate evidence of test-taker’s performance regarding the GED® Science Practices, a subset of the GED® Science Assessment Targets

MethodUse design patterns during item and scenario development to help writers develop a coherent concept that explicitly references:

– The knowledge, skills, and abilities that we are assessing– The types of observations that provide evidence about the KSAs– The features of a task or situation that provides that evidence

3 ECD in GED® Science

Design Pattern in Assessment

Design patterns are ECD tools used to guide the development of test scenarios and associated items

A type of advance organizer for scenario/item writing

Design patterns consist of a series of attributes that reflect assessment argument structure

They lay out a design space for item writers• Choices, connections, examples• Aspects to consider• Methods of increasing or decreasing cognitive difficulty

Benefits of Design Patterns

ECD in GED® Science5

Connect the assessment argument to the items

Can (in the long run) improve efficiency and validity

Help scenario/item writers strengthen the alignment between their scenarios/items and the knowledge and skills they are trying to assess

Support idea generation

Inform the specific aspects of scenarios/items

Design Pattern Attributes

Focal Knowledge, Skills, & Abilities (KSAs)• The primary KSAs targeted by the design patterns• Associated with a Science Practice

Additional KSAs• Other KSAs that may be required for successful performance on

assessment tasks/items

Potential Work Products• Some possible things one could see students doing in response to an

item that would give evidence about the KSAs

Potential Observations• Features of the things students say, do, or make on an item that

constitute the evidence

ECD in GED® Science6

Design Pattern Attributes

Characteristic Features• Aspects of assessment situations/items that are likely to evoke the

desired evidence

Variable Features• Aspects of assessment situations/items that can be varied in order to

shift difficulty or emphasis

Narrative Structure• Type of storyline used as a framework for scenario context

ECD in GED® Science7

Design Patterns in GED® Scenario Writing

Each scenario and scenario-based item should be aligned to/informed by a design pattern when possible

Multiple design patterns can be used to write items in a scenario

Design patterns are housed in SRI’s online PADI Design System“Principled Assessment Design for Inquiry” (PADI)

An 11-step process was developed to guide item writers through the use of PADI, design patterns

Activation chart was designed to connect design patterns to GED® Science Practices

8 ECD in GED® Science

An 11-Step Process for Using Design Patterns

Steps 1 & 2: Become familiar with design patterns, PADI, assignment

Step 3: Brainstorm and research a suitable scenario topic idea

Step 4: Refer to activation chart, connect design pattern KSAs to GED® Science Practices

Step 5: Use design pattern attributes to develop item ideas

Steps 6 & 7: Modify scenario idea based on item ideas and design patterns, ensure alignment of scenario and items

Steps 8-11: Document decisions, obtain approval, write items, and submit, revise, resubmit assignment

9 ECD in GED® Science

Design Pattern Activation Chart

Connects design pattern Focal KSAs to Science Practice indicators

10 ECD in GED® Science

ECD in GED® Science11

Results, Reflections, Lessons Learned

The 11-step process simplified and directed the use of design patterns

The activation chart is a valuable tool for new writers, allowing them to focus on the writing rather than finding an appropriate design pattern

Writers find the design pattern content and attributes to be moderately helpful in writing scenarios and items

Increased usefulness may be gained as writers become more familiar with design patterns and PADI

Additional Information

Technical reports, presentations, and project information

http://ecd.sri.com

http://padi.sri.com