1 Participant Introductions Session: Assessing English Learner Progress and the Quality of English Language Programs Presenters: • Sharon Prestridge, EL Program Coordinator, Mississippi Department of Education • Mark Hansen, Assistant Professor, UCLA-CRESST • Maria Santos, Education Consultant • Pete Goldschmidt, Professor, California State University Northridge Discussant: • Kenji Hakuta, Professor Emeritus, Stanford University
60
Embed
Session: Assessing English Learner Progress and the ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1Participant Introductions
Session: Assessing English Learner Progress and the Quality of English Language Programs
Presenters:
• Sharon Prestridge, EL Program Coordinator, Mississippi Department of Education
• Mark Hansen, Assistant Professor, UCLA-CRESST
• Maria Santos, Education Consultant
• Pete Goldschmidt, Professor, California State University Northridge
Discussant:
• Kenji Hakuta, Professor Emeritus, Stanford University
mdek12.org
June 21, 2021
EL Program Coordinator / Accommodations Coordinator
Sharon Prestridge
Overview of Project - Evaluating English
Language Progress Models: The Sensitivity of
Claims about Progress across State Models
ALL Students Proficient
and Showing Growth in All
Assessed Areas
EVERY Student Graduates
from High School and is Ready
for College and Career
EVERY Child Has Access
to a High-Quality Early
Childhood Program
EVERY School Has Effective
Teachers and Leaders
EVERY Community Effectively
Uses a World-Class Data System to
Improve Student Outcomes
EVERY School and District is
Rated “C” or Higher
1
2
3
4
5
6
3State Board of Education STRATEGIC PLAN GOALS
To create a world-class
educational system that gives
students the knowledge and
skills to be successful in
college and the workforce,
and to flourish as parents
and citizens
VISION
To provide leadership
through the development of
policy and accountability
systems so that all students
are prepared to compete in
the global community
MISSION
Mississippi Department of Education 4
5Project Purpose
The project evaluates English language progress models among
seven participating states to examine the sensitivity of claims
about progress toward English language proficiency and examine
the relationship between model results and the quality of EL
program implementation.
6Competitive Grants for State Assessments (CGSA)
➢ Project – Evaluating English Language Progress Models: The Sensitivity
of Claims about Progress across State Models (a four-year 2.4M grant
project)
• Research – evaluate English language progress models among
participating states
• Development – develop ELP criterion models and an EL program
implementation survey
• Application – Support states in improving EL growth models and in
using ELP assessment data and the EL program implementation
survey to inform school improvement efforts
➢ CCSSO Role – Project Management Partner overseeing the project
state), Washington, and Wisconsin (All states combined serve approximately 462,000
ELs.)
➢ Five partner/consulting organizations:
1. Council of Chief State School Officers (CCSSO)
2. California State University Northridge (CSUN)
3. Center for Research on Evaluation, Standards, and Student Testing (CRESST) at
the University of California Los Angeles
4. Com-Link, LLC
5. Independent Education Consultant
➢ EL Technical Advisory Group: National EL experts and assessment consortium
representatives
8Consultants
Research Design for Evaluating English Language Progress
Models
➢ CSUN: Pete Goldschmidt
Research Design for Developing an EL Program Implementation
Survey
➢ CRESST: Mark Hansen and Zhaopeng Ding
➢ Independent Education Consultant: Maria Santos
Project Evaluation Plan
➢ Com-Link, LLC: Jane Nell Luster
9Project Goals
1. Help states effectively measure English learner growth towards English language proficiency.
2. Help states understand the relationship between English learner progress and EL program implementation.
3. Support participating states in improving their own growth models and in using ELP assessment data to inform school improvement efforts.
4. Inform the field’s understanding of effective measures of growth towards English language proficiency and provide resources that states can customize and use.
10Project Outcomes
1. Disseminate research findings on existing models of measuring and assessing
ELP.
2. Develop new criterion growth models that can improve and broaden the array of
accountability modeling options.
3. Produce recommendations on using growth and ELP indicator results effectively
to monitor programs for ELs.
4. Disseminate the EL Program Implementation Survey and findings to help states
understand how ELP indicator and growth results relate to the EL program
implementation that student receive.
5. Produce a technical report that documents the research design, processes,
analysis procedures, findings, and lessons learned.
1. Help states effectively measure English learner growth towards English language proficiency
2. Help states understand the relationship between English learner progress and EL program implementation
3. Support participating states in refining their own growth models and using ELP assessment data to inform school improvement efforts
4. Inform the field’s understanding of effective measures of growth towards English language proficiency and provide resources that states can customize and use
• What are the ways in which schools can influence the development of Englishlanguage proficiency? What aspects of the school context are relevant toEnglish learners' progress in attaining English language proficiency?
• How can (or how have) these aspects be measured?
1. Collect literature related to education of English learners
• research articles, reports, and books that present or review synthesize research findings regardingeducation of English learners
• USED and state guidance regarding school practices
• data collection instruments including questionnaires, rubrics, checklists, observation tools, interview orfocus group protocols, and document review tools
2. Code the literature
• For research articles, reports, books, and official guidance, coding categories include grade bandsaddressed, topics discussed, input domains (e.g. instruction, curriculum, school climate) investigated,outcome domains (e.g. academic learning, English language proficiency, social emotional learning,graduation) measured
• For instruments: dimensions described/measured, administrators, perspective, purposes (e.g.,compliance review or research/evaluation)
3. Collect/extract and code items/indicators from the instruments.
Policies (Intended)• identification• program models• assessment• program evaluation• support for staff
Leadership/Support• values and beliefs• stakeholder engagement• improvement efforts• professional development• material support• family and community partnership
ELs’ achieve college- and career-readiness
ELs’ progress in attaining English language proficiency
Practices (Enacted)• program model• access to qualified educators• curriculum• instructional practices• assessment practices• access to the full curriculum
• Generate greater understanding of appropriate ways to model students’ English language progress towards proficiency,
• the role ELP assessments play in growth models,
• the impact of model choice on state accountability,
• the impact of model choice on claims about student progress, and
• the relationship between the growth model, ELP indicator results and EL program implementation at schools.
• To support states in monitoring the progress of English Learners, to better understand and utilize progress model results, to better understand program implementation, and to identify the relationships between progress model results and program implementation.
• Process and analyses in this project develops framework for examining the validity of claims about schools based on accountability system results.
TiP 1 TiP 2 TiP 3 TiP 4 TiP 5 TiP 6 TiP 7 TiP 8 TiP 9 TiP 10 TiP 11 TiP 12 TiP 13
IL 1 IL 2 IL 3 IL 4
TiP = Time in Program
IL = Initial Level
HOW DO EXPECTATIONS ALIGN WITH ACTUAL PROGRESS?
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
o1 o2 o3 o4 o5 o6 o7 o8 o9 o10
EFFECT OF INITIAL GRADE LEVEL (IGL) ON OBSERVED PROGRESS
IGL K
Sc
ale
Sc
ore
IGL = Initial Grade Level
EFFECT OF IGL: TRAJECTORIES OF STUDENTS WITH IGL IN K AND FIRST
GRADE.
IGL K Grade 1
TRAJECTORIES ACROSS IGL K TO 11
IGL K Grade 1 Grade 2 Grade 3
Grade 4 Grade 5 Grade 6 Grade 7
Grade 8 Grade 9 Grade 10
EFFECT OF GRADE: TWO ADDITIONAL STATES
K Grade
1
Grade
2
Grade
3
Grade
4
Grade
5
Grade
6
Grade
7
Grade
8
Grade
9
Grade
10
Grade
11
IGL K Grade 1 Grade 2 Grade 3
Grade 4 Grade 5 Grade 6 Grade 7
Grade 8 Grade 9 Grade 10
K Grade
1
Grade
2
Grade
3
Grade
4
Grade
5
Grade
6
Grade
7
Grade
8
Grade
9
Grade
10
Grade
11
IGL K Grade 1 Grade 2 Grade 3 Grade 4 Grade 5
Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
THE EFFECT OF IGL BY TIME IN PROGRAM
TiP 1 TiP 2 TiP 3 TiP 4 TiP 5 TiP 6 TiP 7
IGL K Grade 1 Grade 2 Grade 3
Grade 4 Grade 5 Grade 6 Grade 7
Grade 8 Grade 9 Grade 10
Progress over time is
impacted by IGL
however the effect is
most prevalent in K
and first grade.
The effect size in:
K = 0.2
In 1st = 0.1
In 2nd and up = 0.05.
DERIVING MEANING FROM THE ELPI
• Identifying whether there are systematic issues in progress related to the ELPI.
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Level 1 Level 2 Level 3 Level 4
ELPI = 2
Years in Program
Initial ELPI Level
DERIVING MEANING: COMPARING TWO LEVELS OF ELPII
• ELPI 2 and 4 schools have similar patterns of progress over time and consistent with expectations progress in ELPI 4 schools is generally higher across both initial levels and Time in Program.
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Level 1 Level 2 Level 3 Level 4
ELPI = 2
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Level 1 Level 2 Level 3 Level 4
ELPI = 4
DERIVING MEANING: ARE LOW PERFORMING SCHOOLS SIMPLY
LOWER?• ELPI 1 school is not simply lower progress, but also idiosyncratic patterns of progress.
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Level 1 Level 2 Level 3 Level 4
ELPI = 4
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Level 1 Level 2 Level 3 Level 4
ELPI = 1
Discussion and Q&A
Kenji Hakuta, Professor Emeritus, Stanford University
Sharon Prestridge, Mississippi Dept. of Education
Castañeda's Three-Pronged Test
Source: Callahan, R.M. & Hopkins, M. (2018). Using ESSA to improve secondary English learners’ opportunities to learn through
course taking. Journal of School Leadership, 27(5), 756-767.
Selecting programs based on sound
theory and/or high-quality research
findings
Implementing programs well with sufficient resources
and personnel
Engaging in ongoing evaluation efforts to examine effects and