Jamal Abedi National Center for Research on Evaluation, Standards, and Student Testing UCLA Graduate School of Education & Information Studies November 18, 2004 Psychometric Issues in the ELL Assessment and Special Education Eligibility English Language Learners Struggling to Learn: Emergent Research on Linguistic Differences and Learning Disabilities
39
Embed
Jamal Abedi National Center for Research on Evaluation, Standards, and Student Testing UCLA Graduate School of Education & Information Studies November.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Jamal Abedi
National Center for Research on Evaluation, Standards, and Student Testing
UCLA Graduate School of Education & Information StudiesNovember 18, 2004
Psychometric Issues in the ELL Assessment and Special Education Eligibility
English Language Learners Struggling to Learn:Emergent Research on Linguistic Differences and
Learning Disabilities
Why Should English Language Learners be Assessed?
Goals 2000
Title I and VII of the Improving America’s School Act of 1994 (IASA)
No Child Left Behind Act
Should Schools Test English Language Learners?
Yes
Assessment outcomes may not be valid because their low level English proficiency interferes with content knowledge performance
Test results affect decisions regarding promotion or graduation
They may be inappropriately placed into special educational programs where they receive inappropriate instruction
ELL students may not have received the same curriculum which is assumed for the test
General Problems
English language learners (ELLs) can be placed at a disadvantage because:
Should Schools Test English Language Learners?
YesProblems In Large-Scale Assessment:
Standardized assessment
• Assessment tools in large-scale assessments are usually constructed based on norms that exclude ELL populations
• Research shows major differences between the performance of ELL and non-ELL students on the results of standardized large-scale assessments
• The tests may be biased in favor of non-ELL populations
Performance/alternative assessment
• Such assessments require more language production; thus students with lower language capabilities are at a greater disadvantage
• Scorers may not be familiar with rating ELL performance
Problems
Due to the powerful impact of assessment on instruction, ELL and SWD students’ quality of instruction may be affected
If excluded, they will be dropped out of the accountability picture
Institutions will not be held responsible for their performance in school
They will not be included in state or federal policy decision
Their academic progress, skills, and needs may not be appropriately assessed
Should Schools Test English Language Learners?
No
States with the Highest Proportion of ELL Students
Percentage of Total Student Population:
California 27.0
New Mexico 19.0
Arizona 15.4
Alaska 15.0
Texas 14.0
Nevada 11.8
Florida 10.7
Problems in AYP Reporting: Focus on LEP Students
1. Problems in classification/reclassification of LEP students (moving target subgroup)
2. Measurement quality
3. Low baseline
4. Instability of the LEP subgroup
5. Sparse LEP population
6. LEP cutoff points (Conjunctive vs. Compensatory model)
Site 2 Stanford 9 Sub-scale Reliabilities (1998) Grade 9 Alphas
Familiarity/frequency of non-math vocabulary: unfamiliar or infrequent words changed
census > video gameA certain reference file > Mack’s company
Length of nominals: long nominals shortened last year’s class vice president > vice presidentthe pattern of puppy’s weight gain > the pattern above
Question phrases: complex question phrases changed to simple question words
At which of the following times > Whenwhich is best approximation of the number >
approximately how many
Linguistic Modification Concerns
Conditional clauses: conditionals either replaced with separate sentences or order of conditional and main clause changed If Lee delivers x newspapers > Lee delivers x newspapers
If two batteries in the sample were found to be dead > he found three broken pencils in the sample
Relative clauses: relative clauses either removed or re-cast A report that contains 64 sheets of paper > He needs 64
sheets of paper for each report
Voice of verb phrase: passive verb forms changed to activeThe weights of 3 objects were compared > Sandra
compared the weights of 3 rabbitsIf a marble is taken from the bag > if you take a marble
from the bag
Linguistic Modification cont.
Original:
2. The census showed that three hundred fifty-six thousand, ninety-seven people lived in Middletown. Written as a number, that is:
A. 350,697B. 356,097C. 356,907D. 356,970 Modified:
2. Janet played a video game. Her score was three hundred fifty-six thousand, ninety-seven. Written as number, that is: A. 350,697B. 356,097C. 356,907D. 356,970
Interview StudyTable 1. Student Perceptions Study: First Set (N=19)
Item # Original item chosen Revised item chosen
1 3 16
2 4 15
3 10 9
4 11 8
Table 2. Student Perceptions Study: Second Set (N=17)
Item # Original item chosen Revised item chosen5 3 14
6 4.5a 12.5
7 2 15
8 2 15
Many students indicated that the language in the revised item was easier:
“Well, it makes more sense.”
“It explains better.”
“Because that one’s more confusing.”
“It seems simpler. You get a clear idea of
what they want you to do.”
Issues in the ELL Special Education Eligibility
Issues concerning authenticity of English language Proficiency tests
Issues and problems in identifying students with learning disability in general
Distribution of English language proficiency across ELL/non-ELL student categories
Issues concerning authenticity of English language Proficiency tests
Issues in theoretical bases (discrete point approach, holistic approach, Pragmatic approach)
Issues in content coverage (language proficiency standards)
Issues concerning psychometrics of the assessment
Low relationship between ELL classification categories and English proficiency scores
Issues and problems in identifying students with learning disability in
general A large majority of students with
disabilities fall in learning disability
Validity of identifying students with learning disability is questionable
Distribution of English language proficiency across ELL/non-ELL
student
Most of the existing tests of English proficiency lack enough discrimination power
There is a large number of ELL students perform higher than non-ELL student
The line between ELL and non-ELL on their English proficiency is not a clear line
Reducing the Language Load of Test Items
Reducing unnecessary language complexity of test items helps ELL students (and to some extent SWDs) present a more valid picture of their content knowledge.
The language clarification of test items may be used as a form of accommodation for English language learners.
The results of our research suggest that linguistic complexity of test items may be a significant source of measurement error for ELL students.
Conclusions and Recommendation
1. Classification Issues
Classifications of ELLs and SWDs:
Must be based on multiple criteria that have predictive power for such classificationsThese criteria must be objectively definedMust have sound theoretical and practical basesMust be easily and objectively measurable
Conclusions and Recommendation
2. Assessment Issues
Assessment for ELLs and SWDs:
Must be based on a sound psychometric principlesMust be controlled for all sources of nuisance or confounding variablesMust be free of unnecessary linguistic complexitiesMust include sufficient number of ELLs and SWDs in its development process (field testing, standard setting, etc.)Must be free of biases, such as cultural biasesMust be sensitive to students’ linguistics and cultural needs
3. Issues concerning special education eligibility particularly in placing ELL
students at the lower English language proficiency in the learning/ reading
disability category
There are psychometric issues with the English language proficiency tests
Standardized achievement tests may not provide reliable and valid assessment of ELL students
Reliable and valid measures are needed to distinguish between learning disability and low level of English proficiency
Conclusions and Recommendation
4. Accommodation Issues
Accommodations:
Must be relevant to the subgroups of students Must be effective in reducing the performance gap between accommodated and non-accommodated studentsMust be valid, that is, accommodations should not alter the construct being measuredThe results could be combined with the assessments under standard conditionsMust be feasible in the national and state assessments