California English Language Development Test: Developing the Early Literacy Assessment Components

CALIFORNIA DEPARTMENT OF EDUCATIONJack O’Connell, State Superintendent of Public Instruction

California English Language Development Test:

Developing the Early Literacy Assessment Components

Panel Presentation at the Tenth Annual Accountability Leadership Institute

Burlingame, CADecember 8, 2009

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 2

Section 1 – Introductions, Background, and Policy

Requirements

Lily Roberts, Administrator,California English

Language Development Test (CELDT) Office

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 3

Presentation Overview

• Background and policy requirements• Item development cycle• Guiding principles for test

development and administration• Field test and lessons learned• Development and technical issues• Discussion• Questions and answers

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 4

Background• 2007: State legislature authorized

development of the Kindergarten and grade one (K-1) early literacy assessment

• 2008: Initial item development and expert panel was convened

• 2009: Field test was conducted in March

• 2009-10: Operational testing in all four domains for all grades (K-12)

• 2010: Standard setting will be conducted to recommend performance levels

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 5

Policy Requirements

• Title III, Elementary and Secondary Education Act requires testing in the domains of listening, speaking, reading, and writing in all grades

• California Education Code (EC) Section 60810 defines the purposes for the CELDT:– Identify limited English proficient students – Determine the level of English language

proficiency– Assess progress acquiring the skills of

listening, speaking, reading, and writing

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 6

Reclassification Criteria

• EC Section 313 specifies four criteria for local reclassification decisions, including:– Assessment of English language

proficiency– Performance in basic skills– Teacher evaluation– Parental opinion and consultation

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 7

Section 2 - Initial Item Development

Edynn Sato, WestEd

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 8

2008 Item Development

• Subject matter experts participated in guiding item development

• Conducted two rounds of content and bias reviews

• Developed 142 K-1 reading and writing test questions

• Twelve reading and writing English Language Development (ELD) standards were identified as important and assessable for K-1 by subject matter experts

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 9

Item Development Cycle

• Test specifications are created based on ELD standards

• Guiding principles for item development are established

• Item specifications are created to guide item development

• Item writers and editors receive training

• Items are written and edited

• Graphics are created

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 10

Item Development Cycle (cont.)

• Lead editors and content director review and sign-off on items that are ready for CELDT Office and committee reviews

• Interdepartmental staff reviews items/graphics

• Subject matter experts review items for content

• Subject matter experts review items for bias

• Necessary edits are made

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 11

Guidelines for Item Writing• General Guidelines

– Typical representations– Reflect a range of cultural and ethnic

diversity– Familiar to a range of students

• Specific Guidelines– Specific, detailed, and clear– Contextual detail to enhance

comprehensibility– People in graphics

• Age, gender, ethnicity

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 12

Specific Questions that Need to be Asked during Content Review

• Is the item aligned to the correct ELD Standard?

• Does the item test a knowledge or skill that is appropriate for the assigned domain?

• Is the item aligned to the correct grade span?

• Does the item test for current and worthwhile concepts?

• Does the item stem give a full sense of what is being asked?

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 13

• Are the directions clear?• Does the stem avoid wordiness? • Is the key the only correct answer?• Does the item use distractors that relate

to the stem in the same way as the key?

• Does the item use distractors that are plausible misconceptions and errors?

• Does the item contain clues to the correct answer?

Specific Questions (cont.)

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 14

Specific Questions (cont.)

IF THE ITEM CONTAINS GRAPHICS…

• Is the graphic required to answer the item?

• Does the graphic provide sufficient information needed to answer the item?

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 15

Item Bias and Sensitivity Review

• Ensure that the items avoid stereotype patterns

• Ensure that the items avoid language, symbols, words, phrases, or examples that are biased or potentially offensive, inappropriate, or negative toward any group

• Ensure that the items acknowledge the multicultural nature of our society

• Ensure items do not unfairly advantage or disadvantage any group

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 16

Different Types of Bias

• Bias can refer to age, gender, race, ethnicity, cultural background, social-economic background, religion, disability, or mental condition.

• Bias in test items can be present as Content bias or Language bias.

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 17

Additional development activities:

• Cognitive interviews of students• Interviews of teachers • Small-scale pilot of items

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 18

Section 3 - Field Test and Lessons Learned for

Future CELDT Editions

Caroline Fahmy,Educational Data Systems,

CELDT Test Contractor

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 19

Overview: Lessons Learned K-1 Early Literacy

• Review of test and lessons learned related to:– Spring 2009 field test– 2009-10 Operational test– Modifications to be incorporated into

the 2010-11 operational test

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 20

2009 Field Test (FT)

• 142 items were delivered to the new CELDT contractor in February

• Conducted FT March 16-20, 2009

• Constructed four FT forms with ten reading and ten writing items each

• Administered FT in 37 districts to 2,548 K-1 students in 57 schools across California

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 21

Number of Field Test Participants by Grade

FORM Kinder-garten

Grade One Total

1 274 362 636

2 370 296 666

3 287 322 609

4 347 290 637

Total 1,278 1,270 2,548

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 22

2009-10 Lessons Learned: Test Materials

• Format student book to eliminate distractions and enhance ease of administration

• Provide all specific, test question information in the student book for ease and speed of administration

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 2323

1Examiner Page

Student Page

2 1

2

Page Layout Changes

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 24

2009-10 Lessons Learned:Test Examiner

• Include examples on how to pronounce the letter sounds next to each item script

• Allow examiner to repeat any question, but no prompting allowed

• Provide allowable prompts in the sample practice items in the student test book

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 25

2009-10 Lessons Learned:Student

• Use formatting to minimize distractions

• Use sample item administration as a time to really practice the items

• Do not underestimate students’ English language ability

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 26

2010-11 Lessons Learned: Test Materials

• Student Book and Examiner’s Manuals will include all four domains

• Examiner’s Manual will contain all instructions, questions, and pictures

• Student Book will only contain:– Bubbles for responses– Areas for written responses

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 27

2010-11 Lessons Learned: Test Materials (cont.)

• Student Books will be greatly reduced in size

• Scoring Guide will be combined with Examiner’s Manual (for all grades)

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 28

2010-11 Lessons Learned: Test Examiner

• Move instructions from Student Book to Examiner’s Manual

• Refine the “Acceptable Responses Include” section (based on actual student responses)

• Review the use of pictures in the writing domain

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 29

2010-11 Lessons Learned: Student

• Writing area - modify open box design to:– Help student know where to write– Help administrator and scorer better

interpret upper case and lower case letters– Keep all writing in area clipped for image

scoring

• Use page formatting to accommodate left-handed students in copying words

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 30

Section 4 - Technical Considerationsand Discussion

Richard Duran, University of California, Santa Barbara

Member, CELDT Technical Advisory Group

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 31

K-1 Expert Panel and Technical Advisory Group

• Input on development issues– Guiding principles– Alignment to ELD Standards

• Input on technical issues– Pilot test– Field test– Operational test

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 32

Primary Development Issues

• Guiding Principles and ELD Standards:

Items must:– embody ELD Standards– be based on early literacy theory/research– be meaningful to EL K-1 students– easily administered– scored reliably

• Other Concerns

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 33

Technical Issues• December 2008 pilot test on small K-1 sample of

English learner and English-only students– p-values computed– Cognitive interviews conducted– Screened out some items and revised others– Procedures for administration reviewed

• March 2009 K-1 field test of four forms

- Ten reading and ten writing items per form- Reading items had already been calibrated on

common reading scale- Computed basic test statistics, p-values, point

biserial correlations

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 34

Technical Issues (cont.)

• July 2009 implemented operational test and a linking study for K-1 writing items with a sample of grade two English learners

• January 2010 will conduct a standard setting for K-1 reading and writing performance-level cut scores using a modified Bookmark process

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 35

Discussion

• Achieved alignment of K-1 assessments with state and federal policy and professional testing standards

• Contributed to state and national efforts to better ground English language development assessment and practice

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 36

For More Information

Lily Roberts, Ph.D., Administrator

CELDT Office

Statewide Assessment Division

Telephone: 916-319-0784

E-mail: CELDT@cde.ca.gov

Web: www.cde.ca.gov/ta/tg/el