Thoughts on the Design of an Assessment System for the “Race to the Top” Jim Pellegrino.

Thoughts onthe Design of an

Assessment Systemfor the

“Race to the Top”

Jim Pellegrino

Issues & Challenges

1. Elements of a Comprehensive System

2. Going beyond Standards: Integrating Assessment with Curriculum & Instruction

3. Design Options for K-8 and High School

4. Capitalizing on Affordances of Technology

5. Science an Equal Partner

6. Phasing in Components & Building Capacity

7. Logic Models and a Theory of Action

8. Articulating RttT with ESEA & other Policies

1. Elements of aComprehensive System

Desired end product is a multilevel system Each level fullfills a clear set of functions and has a clear set of

intended users of the assessment information The assessment tools are designed to serve the intended purpose

• Formative, summative or accountability• Design is optimized for function served

The levels are articulated and conceptually coherent They share the same underlying concept of what the targets of

learning are at a given grade level and what the evidence of attainment should be.

They provide information at a “grain size” and on the “time scale” appropriate for translation into action.

Multilevel Assessment System

An Integrated System

Coordinated across levels

Unified by common learning goals

Synchronized by unifying progress

variables

Elements of aComprehensive System (cont)

The system is designed to track progress over time At the individual student level At the aggregrate group level

The system uses tasks, tools, and technologies appropriate to the desired inferences about student achievement Don’t force everything into a fixed testing/task model Use performance tasks, portfolios, projects, fixed- and open-

response tasks as needed

2. Integrate Assessmentwith Curriculum & Instruction

Assessment is part of a coordinated system that includes explicit assumptions about Curriculum and Instruction All three components work in concert and are designed as

complements They are designed together rather than separately

• Curriculum specifies what is to be learned and understood and why• Instruction specifies the features of learning environments -- what

teachers, students and others do -- that are designed to promote the desired forms of learning

• Assessment specifies how the processes and products of teaching and learning will be appraised relative to what was intended for outcomes

Assessment Should not be the “Tail that Wags the Dog”

Assessment

Integrate Assessment with Curriculum & Instruction (cont)

Common Core Standards are a start but not enough Standards are ambiguous and lack detail -- they need to be

translated using “backwards design” and “evidence-centered design” processes

Need to be clear about the Claims one wishes to make about students, the Evidence that would back up those claims, and the Tasks that can provide the critical forms of evidence

There are good examples for how to engage in this ECD process at scale Redesign of the AP science courses and exams College Board science standards for college success

Using an Explicit Domain Modelof Knowledge and Skills

to Coordinate C-I-A in AP

3. Consider Alternative Design Options for K-8 & High School

K-8 should be designed to prepare students for success in high school Need to have a clear, coherent, and cumulative model linking C-I-A

in each major subject-matter area. The conceptual model needs to incorporate learning progressions

and benchmark targets at critical age/grade levels Assessment practices need to promote high levels of performance

(and not undermine them)• Task characteristics should be matched to desired intellectual

performance demands Multiple assessment options may maximize information yield for

differing intended levels of use• Mixture of census testing and matrix sampling may be optimal• Consider periodic vs yearly assessments in core subjects

Consider Alternative Design Options for K-8 & HS (cont)

High School should be designed to prepare students for college or work The same model of testing as K-8 may be inappropriate and sub-

optimal - not all students take same courses at same time Alternatives need to be considered such as assessment closely

linked to curriculum and instruction frameworks in core subject-matter areas

Options for certification of college-readiness and rapid transition to college should be considered in lieu of current “time in school” graduation models

Need flexibility rather than rigidity in considering possible transitions from high school to college or high school to vocational/occupational training

4. Capitalize on the Multiple Affordances of Technology

Makes it possible to tap a broader repertoire of cognitive skills and knowledge Can go beyond conventional practices for item presentation and

conventional stimuli Can implement a range of task designs and item formats Can record and score complex aspects of behavior Functioning models of technology use exist for NAEP and PISA --

basis from which to extrapolate

Can adapt assessments to the needs of different learners and student populations May better enable use of Universal Design principles Can implement adaptive assessment methods

Capitalize on the Multiple Affordances of Technology (cont)

Makes possible the embedding of assessments in learning environments Permits the design and implementation of diagnostic and formative

assessment systems There already exist a number of workable systems for use in

schools in areas of literacy, math and science

Technology can fulfill multiple functions in the design delivery, and use of assessments in multilevel systems that involve multiple users and multiple information needs

Examples of FormativeAssessment Tools

Collecting Reporting Using

Create assessment Manage assessment data Identify resources

Administer assessment Analyze assessment data Identify possible actions

Collect assessment data Create reports Design Professional

Development

Score assessment Distribute reports Deliver Professional Development

Multiple Functions of Technologyin an Assessment System

5. Make Science anEqual Partner

Science at K-8 is in deperate need of improvement The U.S. has no coherent model for what is to be taught and learned in K-8

science International assessments show that science is one of our weakest areas

and it gets worse across time Leaving science out of the system contributes to its second tier status

Allows it to be treated as a “periodic” rather than ongoing and critical instructional issue

Science offers multiple ways to support the development of verbal and quantitative literacies

Much of science is about the process of reasoning from evidence -- doing so requires both qualitative and quantitative reasoning skills

Make Science anEqual Partner (cont)

In a 21st century global environment it makes little sense to ignore the scientific and technical literacy of our students

Many of our choices as citizens involve understanding the relationship between local, national and global actions and consequences

Science achievement at the high school level suffers from the poor preparation of students at K-8

There is little in the way of carryover and cumulative knowledge about science content or process from K-8.

Need to reconsider what HS science needs to be taught and learned given advances in science and technology

6. Phase in Components & Build System Capacity

Consider System Design & Implementation as a 7 to 10 Year Comprehensive Effort and Long-term Investment

Can’t build and implement all the components we need in a short time frame Capacity to design and implement coordinated systems and quality

assessments is not there -- not at the scale of design and implementation needed

Identify key leverage points over different time horizons• 1-3 years; 3-5 years; 5-7+ years

If USED rushes states to “do it all” the result may well be no significant improvement

• Unintended consequence: set assessment back another 10 years like what happened with the mandates from NCLB

Phase in Components & Build System Capacity

Much of the change in the productive use of assessment will require extended training in the use of new tools and systems

A substantial professional development effort will be needed across levels of the system

• Teachers, pirncipals, district and state leadership• Teacher education will need to adapt

Processes for the effective collection and use of assessment information will need to be implemented

New technologies and data systems will need to be created and then accommodated in the business practices of education

The shift from an audit mode for assessment to an assistance mode will take time, effort, and $$$$

7. System Logic Model& Theory of Action

States should be asked to lay out a “Logic Model” for system design, implementation and evaluation

Logic Model forces attention to: (a) existing conditions, (b) resources, (c) inputs, (d) outputs, and (e) proximal and distal outcomes

Not only is there a focus on the elements of the system but most especially the assumed logical and causal relationships amnong them

Model enables monitoring the building of the system and its enactment Model also enables strategies for evaluation of the system along the way

and for adjustment and correction as needed

System Logic Model& Theory of Action

A “Logic Model” forces one to specify the presumed “Theory of Action” Helps make explicit assumptions about how particular components

are supposed to work, who is to be impacted, what the expected consequences should be, and WHY.

In a complex assessment system, it is critical that the theory of action be articulated, especially with regard to how assessment information is to be used to improve outcomes over time - who will use what information and how

Competing theories of action can be made explicit in the system design phase -- choices can be made based on the quality of the evidence and/or argument in favor of adopting one theory in lieu of alternatives

8. Policy Articulation:RttT & ESEA

RttT involves a number of policy choices and it will have multiple implications regarding federal, state, and local regulations and funding Be clear what the policy issues are -- consider the intended as well

as potential unintended consequences of the RFP and awards process as well as all the regulatory elements once implemented

Listen carefully to the advice of “unbiased” groups like the NRC Board on Testing and Assessment on issues of assessment design and use, including the use of Value Added Methodologies for teacher evaluation

Figure out how RttT should align with likely changes in ESEA once reauthorized

Avoid the two having major conceptual and operational conflicts such that ESEA renders the RttT investment moot!

Issues & Challenges Redux1. Focus on the design elements needed for creating a

comprehensive and coherent assessment system2. Integrate assessment with curriculum & instruction: Go

beyond standards to unpack the connections3. Consider purposes and design options for K-8 and High

School C-I-A systems4. Capitalize on the multiple affordances of technology5. Make science an equal partner in the RttT effort6. Phase in components & build human capacity 7. Require states to specify a system Logic Model with an

accompanying Theory of Action8. Think through the articulation of RttT with ESEA & other

education and assessment policies