John F. Olson Assessment Solutions Group CCSSO National Conference on Student Assessment

Getting the Most Value for Your Assessment

Dollar: Cost Analysis for the New

Kentucky Assessment System

John F. Olson Assessment Solutions Group

CCSSO National Conference on Student AssessmentDetroit, MI

June 23, 2010

Overview The ASG Cost Model was used to develop the cost of the

new Kentucky assessment system components. ASG’s report provided the KDE with a realistic estimate

of the cost of the key components of its new 2012 assessment system that can be used for budgeting purposes.

Modeling the costs of the new system using the actual design of the new assessment program will provide an accurate estimate of the costs KDE should expect to see when it bids the program in 2010

Assessment Solutions Group www.assessmentgroup.org

Confidential2

John

do we want to say this to everybody. It infers that the accuracy of our estimates is just ok.

Overview (cont.)

ASG model outputs also allowed the KDE to gain a deeper understanding of the costs of the various components of each assessment

This provides KDE the ability to make trade-offs in assessment program design prior to competitive bidding in order to develop the highest quality and cost-effective assessment for the budgeted dollars.

The ASG Cost Model is a variable input, metric-based calculation output model.

Specific assessment program variables are input to the model and applied against cost factors to derive assessment cost

Several hundred variables are contained in the model. The cost for the entire assessment is built from the

ground up, element by element, function by function. Costs are provided for total assessment, by year, by

content area, by grade, and by functional activity.Assessment Solutions Group www.assessmentgroup.org

Confidential3

Key Program Elements & Assumptions The new 2012 Kentucky assessment system consists of

the following elements: Combined CRT/NRT assessments in

Math grades 3-8, 11 Reading grades 3-8, 10 Science grades 4, 7, 11 Social Studies grades 5, 8, 11

A writing test grades 4, 6, 8, 10, 12

A writing editing & mechanics exam grades 4, 8, 12

Two forms of each assessment will be developed for year 1 (2012) with a new form developed each subsequent year.


Confidential4

Key Program Elements & Assumptions Assessments will be given to students during the last

two weeks of the school year, as specified in the new state law (May – early June).

Test results will be mailed to parents and posted online in August.

For this study ASG calculated: Incremental year zero (2011) costs required to develop and field

test the items to be used in the operational, year one (2012) administration

Ongoing total assessment costs for the years 2012, 2013, and 2014


Confidential5

Year 0 CostsCost by functional area for Year 0 (2011)Note that ASG’s base case assumes year 0 work is priced at a 50% margin, which is

typical of “transition year” programs. Year 0 work also is shown at a 30% margin.

September 28, 2009


Confidential6

50% Margin Total Cost - Yr 0Content Development 1,051,113$ Composition/Pre Press 515,641$ Printing 863,431$ Open Ended Scoring 721,175$ Psychometrics 79,102$ Grand Total 3,230,462$

30% Margin Total Cost - Yr 0Content Development 750,795$ Composition/Pre Press 368,315$ Printing 616,737$ Open Ended Scoring 515,125$ Psychometrics 56,501$ Grand Total 2,307,473$

32.5%

16.0%26.7%

22.3%

2.4%Cost Breakdown - Year Zero

Content Development Composition/Pre Press Printing

Open Ended Scoring Psychometrics

2012 CostsCost for each functional area and cost per student – Year 1

(2012)

September 28, 2009


Confidential7

Total Cost Cost per Student

Content Development 1,651,325$ 3.82$ Composition/Pre Press 378,957$ 0.88$ Printing 1,759,796$ 4.07$ Warehouse/Distribution 287,404$ 0.66$ Freight Costs 1,013,851$ 2.34$ Receiving/Scanning/Machine Scoring 863,741$ 2.00$ Open Ended Scoring 4,696,106$ 10.85$ Score Reporting 295,374$ 0.68$ Quality Assurance 144,690$ 0.33$ Psychometrics 184,152$ 0.43$ Technology Support 493,564$ 1.14$ Program Management 766,309$ 1.77$ Total 12,535,268$ 28.96$

13.2%3.0%

14.0%

2.3%

8.1%

6.9%

37.5%

2.4%1.2%

1.5%

3.9%6.1%

Cost per Student Breakdown

Content Development Composition/Pre Press Printing

Warehouse/Distribution Freight Costs Receiving/Scanning/Machine Scoring

Open Ended Scoring Score Reporting Quality Assurance

Psychometrics Technology Support Program Management

2012 Cost by Content Area

Total cost by content area – Year 1 (2012)

September 28, 2009


Confidential8

$3,612,433

$3,170,401 $1,315,057

$1,249,855

$2,223,060

$964,463

2012 - Cost by Content Area

Reading Math Science Social Studies Writing Editing & Mech

Cost by GradeTotal cost for the Kentucky assessment program by grade (2012 – 2014) Note: grades 4 and 8 are tested more frequently; consumable test books are used

for grade 3

September 28, 2009


Confidential9

2012 2013 2014 TotalTotal Cost per GradeGrade 3 1,290,123$ 1,311,013$ 1,363,453$ 3,964,589$ Grade 4 2,085,552$ 2,103,288$ 2,187,420$ 6,376,260$ Grade 5 1,379,098$ 1,388,449$ 1,443,987$ 4,211,534$ Grade 6 1,388,489$ 1,405,356$ 1,461,570$ 4,255,415$ Grade 7 1,363,475$ 1,372,070$ 1,426,953$ 4,162,498$ Grade 8 2,183,805$ 2,205,645$ 2,293,870$ 6,683,320$ Grade 9 -$ -$ -$ -$ Grade 10 956,381$ 971,182$ 1,010,029$ 2,937,592$ Grade 11 1,202,516$ 1,205,976$ 1,254,216$ 3,662,708$ Grade 12 685,829$ 694,995$ 722,795$ 2,103,618$ Grand Total 12,535,268$ 12,657,974$ 13,164,292$ 38,357,534$

Cost by Content Area & Grade

Costs for the Kentucky assessment program by content area & grade for 2012 – 2014.

September 28, 2009


Confidential10

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Total

2012Reading 679,265$ 489,427$ 495,275$ 505,919$ 474,068$ 505,998$ -$ 462,480$ -$ -$ 3,612,433$ Math 610,858 416,430 453,523 426,915 442,961 421,675 - - 398,039 - 3,170,401 Science - 458,430 - - 446,445 - - - 410,182 - 1,315,057 Social Studies - - 430,300 - - 425,260 - - 394,295 - 1,249,855 Writing - 394,353 - 455,656 - 478,598 - 493,901 - 400,552 2,223,060 Editing & Mech - 326,913 - - - 352,274 - - - 285,277 964,463 Grand Total 1,290,123$ 2,085,552$ 1,379,098$ 1,388,489$ 1,363,475$ 2,183,805$ -$ 956,381$ 1,202,516$ 685,829$ 12,535,268$

2013Reading 691,078$ 493,646$ 499,815$ 510,884$ 477,717$ 510,924$ -$ 465,683$ -$ -$ 3,649,747$ Math 619,934 417,730 456,393 428,720 445,365 423,227 - - 399,081 - 3,190,451 Science - 461,409 - - 448,988 - - - 411,709 - 1,322,107 Social Studies - - 432,241 - - 426,956 - - 395,187 - 1,254,384 Writing - 401,909 - 465,751 - 489,567 - 505,499 - 409,051 2,271,776 Editing & Mech - 328,594 - - - 354,970 - - - 285,945 969,509 Grand Total 1,311,013 2,103,288 1,388,449 1,405,356 1,372,070 2,205,645 - 971,182 1,205,976 694,995 12,657,974

2014Reading 718,721$ 513,392$ 519,808$ 531,320$ 496,825$ 531,361$ -$ 484,310$ -$ -$ 3,795,737$ Math 644,732 434,439 474,649 445,869 463,180 440,156 - - 415,044 - 3,318,069 Science - 479,865 - - 466,948 - - - 428,178 - 1,374,991 Social Studies - - 449,530 - - 444,035 - - 410,994 - 1,304,559 Writing - 417,985 - 484,381 - 509,150 - 525,719 - 425,413 2,362,647 Editing & Mech - 341,738 - - - 369,169 - - - 297,382 1,008,290 Grand Total 1,363,453 2,187,420 1,443,987 1,461,570 1,426,953 2,293,870 - 1,010,029 1,254,216 722,795 13,164,292

Total Program CostsTotal program costs for 2011 - 2014

September 28, 2009


Confidential11

Program Costs 2011 2012 2013 2014 Total

Base Cost 1,242,485$ 7,121,851$ 7,161,759$ 7,448,229$ 22,974,325$

Overhead 372,746$ 1,957,646$ 1,982,198$ 2,061,486$ 6,374,076$

Margin* 1,615,231$ 3,455,771$ 3,514,016$ 3,654,577$ 12,239,595$

Total 3,230,462$ 12,535,268$ 12,657,974$ 13,164,292$ 41,587,996$

*Margin rate is 50% for 2011 and 30% in subsequent years. The exception is travel and NRT licensing, which are set at 15%

55.2%

15.3%

29.4%

Total Program Cost % Split: 2011 - 2014

Base Cost Overhead Margin*

Total Costs and Key Metrics by YearTotal program costs and key metrics by year for 2012 - 2014

September 28, 2009


Confidential12

Functional Area Cost Item Unit of Measure Total Cost Total Units Cost per Unit Total Cost Total Units Cost per Unit Total Cost Total Units Cost per UnitContent Development Item Acquisition Items Acquired 4,524$ 158 28.56$ 4,705$ 158 29.70$ 4,893$ 158 30.89$

Permission Costs Permissions 135,386$ 65 2,089.29$ 140,801$ 65 2,172.86$ 146,433$ 65 2,259.77$ Item Development Total Items Developed 325,697$ 158 2,056.17$ 245,897$ 158 1,552.38$ 255,732$ 158 1,614.47$ Art Costs Art Pieces 14,295$ 128 111.75$ 14,866$ 128 116.22$ 15,461$ 128 120.86$ Item Banking NA 17,335$ NA NA 18,028$ NA NA 18,750$ NA NADevelopment Travel NA 654,733$ NA NA 512,795$ NA NA 533,307$ NA NANRT Licensing NCOunt 499,356$ 390,800 1.28$ 519,330$ 390,800 1.33$ 540,103$ 390,800 1.38$

Composition/Pre-Press Total Composition Costs Pages Composed 378,957$ 6,700 56.56$ 394,115$ 6,700 58.82$ 409,880$ 6,700 61.18$

Printing Total Printing Costs Bound Units 1,759,796$ 3,203,741 0.55$ 1,830,188$ 3,203,741 0.57$ 1,903,395$ 3,203,741 0.59$

Warehouse/Distribution Total Warehouse Costs Pounds Shipped 287,404$ 383,502 0.75$ 298,900$ 383,502 0.78$ 310,856$ 383,502 0.81$ Freight Out Cost Pounds Shipped 427,331$ 383,502 1.11$ 444,424$ 383,502 1.16$ 462,201$ 383,502 1.21$ Scoring Freight In Pounds Returned 586,520$ 350,106 1.68$ 609,981$ 350,106 1.74$ 634,380$ 350,106 1.81$

Receiving/Order Entry Documents 38,778$ 1,356,100 0.03$ 40,329$ 1,356,100 0.03$ 41,942$ 1,356,100 0.03$ Document Prep/Staging Documents 468,487$ 1,356,100 0.35$ 487,227$ 1,356,100 0.36$ 506,716$ 1,356,100 0.37$ Image Scanning Sheets Scanned 95,804$ 6,512,200 0.01$ 99,636$ 6,512,200 0.02$ 103,622$ 6,512,200 0.02$ Document Editing N-Count 260,671$ 432,800 0.60$ 271,098$ 432,800 0.63$ 281,942$ 432,800 0.65$

Open Ended Scoring Total Cost Scores 4,696,106$ 6,511,586 0.72$ 4,883,950$ 6,511,586 0.75$ 5,079,308$ 6,511,586 0.78$

Score Reporting Total Cost Pages Printed 295,374$ 1,731,200 0.17$ 307,189$ 1,731,200 0.18$ 319,477$ 1,731,200 0.18$

All Areas Total Program Cost N-Count 12,535,268$ 432,800 28.96$ 12,657,974$ 432,800 29.25$ 13,164,292$ 432,800 30.42$

2014

Receiving/Scanning/Machine Scoring

2012 2013

Summary of Impact of Options/Changes in Costs

September 28, 2009 13


Confidential

Chart shows cost savings or increases related to various options/changes to the program for 2012 - 2014

Summary, Conclusions, & Recommendations ASG performed a special study analyzing the costs of

the new Kentucky 2012 assessment system Year 0 (2011) activities – item development and field testing Years 1 – 3 (2012-2014) – full administration of new assessment

A vendor performing efficiently is expected to price the year 0 activities at $3.2M and the total program costs for years 2012-2014 at $12.5M, $12.7M, and $13.2M respectively

Consistent with other NCLB state programs, the key costs are in the item development, scoring, and printing areas

ASG performed a number of scenarios to consider what could change the cost picture significantly

September 28, 2009


Confidential14

Summary, Conclusions, & Recommendations ASG’s analysis finds the total cost and cost per student

of the new Kentucky assessment system appear to be in a reasonable range for a high stakes, NCLB-type state assessment.

Metrics and benchmarks used in the analysis represent those of an efficient contractor realistically bidding on the program

The proposed item release rate and use of NRT components require only one form per year which significantly controls costs.

September 28, 2009


Confidential15

John

not sure I would say this -- other states may take affront to this stmt since they may think thay have a more comprehensive asmt (and many do). I suggest deleting it.

Summary, Conclusions, & Recommendations ASG recommends that the KDE consider ways to

improve the cost-effectiveness and efficiency of the new assessment system in the future.

When developing the RFP for the new Kentucky assessment program it will be critical to have a clearly written proposal that calls for the vendor to submit the metrics and specifications outlining their understanding of the program requirements.

September 28, 2009 16


Confidential

John F. Olson Assessment Solutions Group CCSSO National Conference on Student Assessment

Documents

new assessment program

costeffective assessment

assessment dollar

assessment costseveral

entire assessment

assessment program design

new system

subsequent year