Design of the 2011 Design of the 2011 NAEP-TIMSS Linking NAEP-TIMSS Linking Study Study Andrew Kolstad Andrew Kolstad Senior Technical Advisor to the Assessment Division Senior Technical Advisor to the Assessment Division National Center for Education Statistics National Center for Education Statistics
16
Embed
Design of the 2011 NAEP- TIMSS Linking Study Andrew Kolstad Senior Technical Advisor to the Assessment Division National Center for Education Statistics.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Design of the 2011 NAEP-Design of the 2011 NAEP-TIMSS Linking Study TIMSS Linking Study
Andrew KolstadAndrew KolstadSenior Technical Advisor to the Assessment DivisionSenior Technical Advisor to the Assessment Division
National Center for Education StatisticsNational Center for Education Statistics
For states and school districts, to project TIMSS mathematics and science scores from performance on NAEP (without the cost of participating in TIMSS)
2
Goals of the study in 2011Goals of the study in 2011Goals of the study in 2011Goals of the study in 2011
Johnson and Cohen’s projection linkage to 2000 grade 8 state NAEP mathematics and science• A linking sample of students was assessed by NAEP in the
winter of 2000 and again by TIMSS in the spring of 2000
• TIMSS-equivalent mathematics and science scores predicted from NAEP state scores using “statistical projection” (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores)
• Evaluated with grade 8 TIMSS in twelve states in 1999o The performance differed: the actual grade 8 means generally fell above
• International grade 8 TIMSS-equivalent scores of NAEP achievement level cut scores projected using “statistical moderation” (no student took both assessments)
• Based on national distributions of NAEP 2000 mathematics and science scores matched to U.S. score distributions on o TIMSS 1999 assessment
o TIMSS 2003 assessment
• Result: Estimated percentage of basic, proficient, and advanced students in each country that participated in the 1999 TIMSS and 2003 TIMSS studies
5
Improving on previous linking designs Improving on previous linking designs
How the current linking study differs
1. Both NAEP and TIMSS in 2011, the same year this time
2. Two linking samples of students; no student assessed in both winter and spring
3. Two booklet designs; one looks like NAEP in the winter and the other looks like TIMSS in the spring
4. Projected TIMSS mathematics and science scores predicted from NAEP state scores using “statistical projection”
o (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores)
5. Evaluated with grade 8 TIMSS in eight states in 20116
Components in linking NAEP & TIMSSComponents in linking NAEP & TIMSS
1. NAEP state-level science and mathematics data needed as a basis for projecting TIMSS mathematics and science scores○ For the first time in NAEP history, all 50 states volunteered to
participate in a voluntary state NAEP
2. TIMSS state-level grade 8 data needed to check the validity of the linking function
3. Linking samples of students will provide responses to items from both assessments; needed to estimate relationships○ No students will take NAEP in the winter and TIMSS in the spring, to
avoid the decline in scores found in the previous study
7
Booklet designs for NAEP & TIMSSBooklet designs for NAEP & TIMSS
8
Main study booklets
NAEP first block second blockBQ1
BQ2
TIMSS first block second block Brk third block fourth blockBrk BQ
1. Administer braided booklets in both testing windowso NAEP in Winter (January to March, 2011) o TIMSS in Spring (April to June, 2011)
2. Select national samples independent of the main NAEP and TIMSS sampleso 10,000 students in NAEP in Winter (half science, half math) o 7,500 students in TIMSS in Spring (half science, half math)
3. Collect the data using NAEP field staff in winter and TIMSS field staff in spring (Westat is the contractor for both)
4. Score the items using the same scoring staff with the same training and QC procedures that NAEP and TIMSS use
10
Booklet Designs for NAEP-TIMSS Linking Studies
NAEP window first block second block
BQ1
BQ2
first block second block
BQ1
BQ2
TIMSS window first block second block
Brk third block fourth block BQ
first block second block Brk third block fourth block BQ
Assessment time 5 10 15 20 25 30 35 40 45
50 55 60 65 70 75 80 85 90 95
100…
115
Legend
NAEP blocks
TIMSS blocks
Booklet designs for braided studyBooklet designs for braided study
11
NAEP-window booklets contain either science or mathematics blocks, but not both
• TIMSS-window booklets contain both science and mathematics TIMSS blocks
• TIMSS-window booklets contain three blocks of science or math, plus one block of NAEP
Example booklet configurations in the TIMSS window
TIMSS M01 NAEP M01 TIMSS S13 TIMSS M03
TIMSS S02 NAEP S01 TIMSS S05 TIMSS M03
Mathematics
Science
TIMSS Math blocks
NAEP Math block
TIMSS Science block
TIMSS Science blocksTIMSS Math block
NAEP Science block
Block composition of braided bookletsBlock composition of braided booklets
12
TIMSS at the state levelTIMSS at the state level
• Planned as a validity check on the NAEP projection methodology
• Administered as part of regular, national TIMSS• Funded by NCES, not by states • 29 states volunteered;
○ 8 states will participate○ MA, MN, AL, CA, CT, IN, CO, & NC
13
NAEP at the state levelNAEP at the state level
• A state-level grade 8 mathematics assessment was already planned
• All states have volunteered to participate in the new 2011 NAEP science assessment
• Science will not be a regular NAEP state-level assessment with full reporting (no urban districts)
• The braided linking samples are national only, with a very small sample in any given state
14
Timeline for NAEP-TIMSS linkingTimeline for NAEP-TIMSS linking
• Release of the national and state TIMSS results is scheduled for December 2012
• Linking results will be reported a few months after the main TIMSS release
15
Benefits of the proposed approachBenefits of the proposed approach
• Avoids interference with the conduct of regular operational assessments
• Provides a NAEP-TIMSS link through common items taken by the same student
• Measures the amount of growth during the period from winter to spring
• Provides states with international benchmarks in science and mathematics at grade 8, without the cost of conducting TIMSS at the state level