To Consortia, or not to Consortia CCSSO National Conference on Student Assessment June 23, 2010

Getting the Most Value for Your Assessment Dollar – Designing Adapting and Maintaining Quality Assessment Programs During Tough Economic Times

To Consortia, or not to Consortia

CCSSO National Conference on Student Assessment

June 23, 2010

June 23, 2010 www.assessmentgroup.org 2

Joining a Consortium

Implementing a new, innovative assessment program in a consortium as a way to save costs. Or,

maintaining a current program without having to make drastic cuts

Is it doable? Can a consortia of states implement a new

assessment at a significantly reduced cost than a single state acting alone?

How large does the consortia need to be? Where are the cost savings opportunities?

Joining a State Assessment Consortium

Joining a state assessment consortium can have its advantages but. . . Requires a lot of planning, coordination

& desire Several successful examples:

NECAP WIDA Achieve Algebra 2 PARCC & SBAC (responses due today)June 23, 2010 3

Stanford/Nellie Mae StudyPurpose of the study was to see if it is

possible to create an affordable “high quality” assessment

Step one – Model a current typical assessment in ELA and Math – Cost $19-$20

Step two – Model a high quality assessment for the same state – Cost $55-$56 a student

Step three – Implement several cost savings strategies

June 23, 2010Assessment Solutions Group www.assessmentgroup.org 4

Cost Reduction Strategies Participation in a consortium

Looked at 10, 20 and 30 state sizes Cost reduction - $15 per student

Uses of technology for online test delivery, distributed human scoring of some of the open-ended items, and automated scoring for certain constructed response items

Together, these innovations account for cost savings of about $3 to $4 per student

Likely to account for more as efficiencies are developed in programming and using technology for these purposes

Two approaches to the use of teacher-moderated scoring. Teacher-moderated scoring can net both substantial cost reductions as well as potential professional development benefits. We used two different models for teacher-moderated scoring

5

Cost Reduction Strategies Two different models for teacher-moderated scoring:

Professional development model - no additional teacher compensation beyond that supported by the state or district for normal professional development days (NY Regents)

Stipend model - assume a $125/day stipend for teachers to score the performance items.

Note: teachers were assumed to score all performance items in a distributed scoring model

These strategies for using teachers as scorers reduce costs by an additional $10 to $20 per pupil (depending on whether teachers are engaged as part of professional development or are paid)

Adopting all cost reduction strategies while paying teachers a $125/day stipend to score all performance tasks results in an assessment cost of $21

June 23, 2010Assessment Solutions Group www.assessmentgroup.org 6

Consortia SizeHow big do you have to be?

Stanford/Nellie Mae study found that 80% of the cost benefits of joining a consortium are realized at the 10 state size.

Rough estimate is that a 5 state consortium could achieve 75%+ of the cost savings of a 10 state consortia

Perhaps $3 - $6 per student $2.7 M/year for the average sized state (600K students)

June 23, 2010 7

Where are the Cost Savings?

Big cost savings opportunity in development Largely a fixed cost function Increase in forms cost partially offsets the savings

Other fixed cost functions such as IT, Quality Assurance and Psychometrics provide savings

Even functions that are largely variable in nature also have a fixed cost component

Some functions like program management allow for economies of scale

June 23, 2010 www.assessmentgroup.org 8

Assessment Costs by Consortium Size

June 23, 2010 9

OnlineNumber of States 1 10 20 30 10

Functional AreaContent Development 4.00$ 0.60$ 0.30$ 0.19$ 0.60$

Paper Based Functions: Printing 2.25$ 1.91$ 1.84$ 1.76$ Warehouse & Distribution 1.97$ 1.82$ 1.77$ 1.70$ Receiving/Scanning/Mach. Scoring 1.85$ 1.71$ 1.66$ 1.60$ Subtotal Paper Based Functions 6.07$ 5.44$ 5.27$ 5.06$

On-line Delivery 3.70$ Constructed Response Scoring 8.86$ 6.99$ 6.74$ 6.46$ 6.99$ Performance Event Scoring 33.45$ 27.43$ 26.61$ 25.53$ 27.43$ Score Reporting 0.76$ 0.71$ 0.69$ 0.66$ 0.71$ Program Management 1.10$ 1.02$ 0.95$ 0.86$ 1.02$ QA/IT/Psychometrics 1.45$ 0.23$ 0.11$ 0.08$ 0.23$ Total 55.69$ 42.42$ 40.67$ 38.84$ 40.68$

PPT

Cost Per Student For High Quality Assessment

Where are the Cost Savings?

Consortia size can make assessment technology more affordable Online test delivery (CBT and CAT) Artificial intelligence scoring of CRs

More states/students more bargaining power

A common assessment with common standards and operational methods s/b more efficient Need to weigh this against potential additional

collaboration costs and risks

June 23, 2010 10

PARCC & SBAC Support We recently assisted both consortia in

preparing their cost estimates for the NIA responses

Both consortia had innovative ideas for new assessments and a wide variety of design and operational decisions to make

Each idea/design choice came with unique cost implications

June 23, 2010 www.assessmentgroup.org 11

PARCC & SBAC Support

Initially, each consortia’s design was deemed too expensive in both the operational and ongoing periods. Each needed adjustments:

The number of choices and variables can be daunting as there are many variables and moving parts

Ultimately, each consortia created innovative assessment systems with the designs they wanted

June 23, 2010 12

Assessment Design Decision Tree

Delivery Method Paper based Computer (linear or CAT) Mixed (both CBT and PPT)

Assessment Types Summative, through course summative Interim/benchmark, End of Course, Formative Domains, special populations

Indicates a major cost element for either PARCC or SBAC

June 23, 2010 13

Decisions and Cost Variables (cont.)

Development Types of items (SR, CR, Computer enhanced,

PE, PT) Mix of item types Number of forms, CAT algorithm (750-1000

items per grade), number of attempts Release rates (by item type) Breach form (develop?, print?) Grades/domains tested Item bank development

June 23, 2010 14

Decisions and Cost Variables

Paper based testing/cutover to CBT How long to cut over (operating in both

modes is very expensive)? Different production strategies

Minimize print page “signatures” Use of color (B/W, grey scale, 4 color) Breach form (print?) Security measures (# of forms, labels, seals,

student ID)June 23, 2010 www.assessmentgroup.org 15

Decisions and Cost Variables Logistics

Transportation mode (ground, air) Carrier selection Ship from/to locations (consolidated

shipping) Meetings and Travel (online vs. live)

Scoring Computer vs. Human (incl. scanning and

editing)June 23, 2010 16

Design Decisions & Costs

Scoring (cont.) Human Method (teacher or 3rd party)

Holistic vs. analytic scoring Requires a lot of work to develop innovative items that

can be scored in a timely manner Alternatively, a test design where these items are

scored during a classroom period may make sense (PEs) AI scoring for open ended items

Math vs. ELA Items requiring inference can’t easily be scored using AI System training fees (fixed cost); per score costs

June 23, 2010 17

Design Decisions & Costs Open-Ended Scoring (cont.)

Double scoring/Read behind rates (by grade)

Distributed vs. on-site Reporting

Paper vs. online reporting Number and complexity of reports

June 23, 2010 www.assessmentgroup.org 18

Conclusion Even a small consortium of states can achieve

significant reductions in assessment cost Such a strategy can be useful in developing a new,

high quality assessment or maintaining a current one during times of budgetary stress

Participating in a consortium also allows for the implementation of innovative technologies that can improve assessment quality and reduce costs

Teacher scoring of open-ended items is critical for implementing a high quality assessment

There are a myriad of design and operational decisions that have significant cost impacts

19

Conclusion“You can’t always get what

you want; but if you try sometime you just might find you get what you need.”

- Mick Jagger

www.assessmentgroup.org 20

Questions? Barry Topol

[email protected] John Olson [email protected] Ed Roeber [email protected]

June 23, 2010 www.assessmentgroup.org 21