Thoughts on the Design of an Assessment System for the “Race to the Top” Jim Pellegrino.
Post on 21-Dec-2015
219 Views
Preview:
Transcript
Issues & Challenges
1. Elements of a Comprehensive System
2. Going beyond Standards: Integrating Assessment with Curriculum & Instruction
3. Design Options for K-8 and High School
4. Capitalizing on Affordances of Technology
5. Science an Equal Partner
6. Phasing in Components & Building Capacity
7. Logic Models and a Theory of Action
8. Articulating RttT with ESEA & other Policies
1. Elements of aComprehensive System
Desired end product is a multilevel system Each level fullfills a clear set of functions and has a clear set of
intended users of the assessment information The assessment tools are designed to serve the intended purpose
• Formative, summative or accountability• Design is optimized for function served
The levels are articulated and conceptually coherent They share the same underlying concept of what the targets of
learning are at a given grade level and what the evidence of attainment should be.
They provide information at a “grain size” and on the “time scale” appropriate for translation into action.
Multilevel Assessment System
An Integrated System
Coordinated across levels
Unified by common learning goals
Synchronized by unifying progress
variables
Elements of aComprehensive System (cont)
The system is designed to track progress over time At the individual student level At the aggregrate group level
The system uses tasks, tools, and technologies appropriate to the desired inferences about student achievement Don’t force everything into a fixed testing/task model Use performance tasks, portfolios, projects, fixed- and open-
response tasks as needed
2. Integrate Assessmentwith Curriculum & Instruction
Assessment is part of a coordinated system that includes explicit assumptions about Curriculum and Instruction All three components work in concert and are designed as
complements They are designed together rather than separately
• Curriculum specifies what is to be learned and understood and why• Instruction specifies the features of learning environments -- what
teachers, students and others do -- that are designed to promote the desired forms of learning
• Assessment specifies how the processes and products of teaching and learning will be appraised relative to what was intended for outcomes
Integrate Assessment with Curriculum & Instruction (cont)
Common Core Standards are a start but not enough Standards are ambiguous and lack detail -- they need to be
translated using “backwards design” and “evidence-centered design” processes
Need to be clear about the Claims one wishes to make about students, the Evidence that would back up those claims, and the Tasks that can provide the critical forms of evidence
There are good examples for how to engage in this ECD process at scale Redesign of the AP science courses and exams College Board science standards for college success
3. Consider Alternative Design Options for K-8 & High School
K-8 should be designed to prepare students for success in high school Need to have a clear, coherent, and cumulative model linking C-I-A
in each major subject-matter area. The conceptual model needs to incorporate learning progressions
and benchmark targets at critical age/grade levels Assessment practices need to promote high levels of performance
(and not undermine them)• Task characteristics should be matched to desired intellectual
performance demands Multiple assessment options may maximize information yield for
differing intended levels of use• Mixture of census testing and matrix sampling may be optimal• Consider periodic vs yearly assessments in core subjects
Consider Alternative Design Options for K-8 & HS (cont)
High School should be designed to prepare students for college or work The same model of testing as K-8 may be inappropriate and sub-
optimal - not all students take same courses at same time Alternatives need to be considered such as assessment closely
linked to curriculum and instruction frameworks in core subject-matter areas
Options for certification of college-readiness and rapid transition to college should be considered in lieu of current “time in school” graduation models
Need flexibility rather than rigidity in considering possible transitions from high school to college or high school to vocational/occupational training
4. Capitalize on the Multiple Affordances of Technology
Makes it possible to tap a broader repertoire of cognitive skills and knowledge Can go beyond conventional practices for item presentation and
conventional stimuli Can implement a range of task designs and item formats Can record and score complex aspects of behavior Functioning models of technology use exist for NAEP and PISA --
basis from which to extrapolate
Can adapt assessments to the needs of different learners and student populations May better enable use of Universal Design principles Can implement adaptive assessment methods
Capitalize on the Multiple Affordances of Technology (cont)
Makes possible the embedding of assessments in learning environments Permits the design and implementation of diagnostic and formative
assessment systems There already exist a number of workable systems for use in
schools in areas of literacy, math and science
Technology can fulfill multiple functions in the design delivery, and use of assessments in multilevel systems that involve multiple users and multiple information needs
Collecting Reporting Using
Create assessment Manage assessment data Identify resources
Administer assessment Analyze assessment data Identify possible actions
Collect assessment data Create reports Design Professional
Development
Score assessment Distribute reports Deliver Professional Development
Multiple Functions of Technologyin an Assessment System
5. Make Science anEqual Partner
Science at K-8 is in deperate need of improvement The U.S. has no coherent model for what is to be taught and learned in K-8
science International assessments show that science is one of our weakest areas
and it gets worse across time Leaving science out of the system contributes to its second tier status
Allows it to be treated as a “periodic” rather than ongoing and critical instructional issue
Science offers multiple ways to support the development of verbal and quantitative literacies
Much of science is about the process of reasoning from evidence -- doing so requires both qualitative and quantitative reasoning skills
Make Science anEqual Partner (cont)
In a 21st century global environment it makes little sense to ignore the scientific and technical literacy of our students
Many of our choices as citizens involve understanding the relationship between local, national and global actions and consequences
Science achievement at the high school level suffers from the poor preparation of students at K-8
There is little in the way of carryover and cumulative knowledge about science content or process from K-8.
Need to reconsider what HS science needs to be taught and learned given advances in science and technology
6. Phase in Components & Build System Capacity
Consider System Design & Implementation as a 7 to 10 Year Comprehensive Effort and Long-term Investment
Can’t build and implement all the components we need in a short time frame Capacity to design and implement coordinated systems and quality
assessments is not there -- not at the scale of design and implementation needed
Identify key leverage points over different time horizons• 1-3 years; 3-5 years; 5-7+ years
If USED rushes states to “do it all” the result may well be no significant improvement
• Unintended consequence: set assessment back another 10 years like what happened with the mandates from NCLB
Phase in Components & Build System Capacity
Much of the change in the productive use of assessment will require extended training in the use of new tools and systems
A substantial professional development effort will be needed across levels of the system
• Teachers, pirncipals, district and state leadership• Teacher education will need to adapt
Processes for the effective collection and use of assessment information will need to be implemented
New technologies and data systems will need to be created and then accommodated in the business practices of education
The shift from an audit mode for assessment to an assistance mode will take time, effort, and $$$$
7. System Logic Model& Theory of Action
States should be asked to lay out a “Logic Model” for system design, implementation and evaluation
Logic Model forces attention to: (a) existing conditions, (b) resources, (c) inputs, (d) outputs, and (e) proximal and distal outcomes
Not only is there a focus on the elements of the system but most especially the assumed logical and causal relationships amnong them
Model enables monitoring the building of the system and its enactment Model also enables strategies for evaluation of the system along the way
and for adjustment and correction as needed
System Logic Model& Theory of Action
A “Logic Model” forces one to specify the presumed “Theory of Action” Helps make explicit assumptions about how particular components
are supposed to work, who is to be impacted, what the expected consequences should be, and WHY.
In a complex assessment system, it is critical that the theory of action be articulated, especially with regard to how assessment information is to be used to improve outcomes over time - who will use what information and how
Competing theories of action can be made explicit in the system design phase -- choices can be made based on the quality of the evidence and/or argument in favor of adopting one theory in lieu of alternatives
8. Policy Articulation:RttT & ESEA
RttT involves a number of policy choices and it will have multiple implications regarding federal, state, and local regulations and funding Be clear what the policy issues are -- consider the intended as well
as potential unintended consequences of the RFP and awards process as well as all the regulatory elements once implemented
Listen carefully to the advice of “unbiased” groups like the NRC Board on Testing and Assessment on issues of assessment design and use, including the use of Value Added Methodologies for teacher evaluation
Figure out how RttT should align with likely changes in ESEA once reauthorized
Avoid the two having major conceptual and operational conflicts such that ESEA renders the RttT investment moot!
Issues & Challenges Redux1. Focus on the design elements needed for creating a
comprehensive and coherent assessment system2. Integrate assessment with curriculum & instruction: Go
beyond standards to unpack the connections3. Consider purposes and design options for K-8 and High
School C-I-A systems4. Capitalize on the multiple affordances of technology5. Make science an equal partner in the RttT effort6. Phase in components & build human capacity 7. Require states to specify a system Logic Model with an
accompanying Theory of Action8. Think through the articulation of RttT with ESEA & other
education and assessment policies
top related