Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs August 2013 Massachusetts Department of Elementary and Secondary Education 75 Pleasant Street, Malden, MA 02148-4906 Phone 781-338-3000 TTY: N.E.T. Relay 800-439-2370 www.doe.mass.edu
34
Embed
Bridge Study Linking the Results of 2012 MEPA and … · Web viewBridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs August 2013 Massachusetts Department of Elementary
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs
August 2013
Massachusetts Department of Elementary and Secondary Education75 Pleasant Street, Malden, MA 02148-4906Phone 781-338-3000 TTY: N.E.T. Relay 800-439-2370www.doe.mass.edu
This document was prepared by the Massachusetts Department of Elementary and Secondary Education
Mitchell D. Chester, Ed.D.Commissioner
The Massachusetts Department of Elementary and Secondary Education, an affirmative action employer, is committed to ensuring that all of its programs and facilities are accessible to all members of the public.
We do not discriminate on the basis of age, color, disability, national origin, race, religion, sex, gender identity, or sexual orientation.
Inquiries regarding the Department’s compliance with Title IX and other civil rights laws may be directed to theHuman Resources Director, 75 Pleasant St., Malden, MA 02148-4906. Phone: 781-338-6105.
Equipercentile Linking Method Used to Bridge ACCESS for ELLs and MEPA Results............................................................................................................................2
Assumptions for Conducting the Bridge Study..........................................................3
Description of the Student Sample...............................................................................3
AMAO: Attainment of English Proficiency...................................................................6
AMAO: Progress in Learning English..........................................................................6
Appendix: ACCESS and MEPA Concordance Tables................................................8
IntroductionThe Department of Elementary and Secondary Education (“the Department”) administered the Assessing Comprehension and Communication in English State-to-State for English Language Learners (ACCESS for ELLs) assessments for the first time in January 2013. Developed by the WIDA Consortium at the University of Wisconsin, ACCESS for ELLs is replacing the Massachusetts English Proficiency Assessment (MEPA) as the state’s annual assessment of English language proficiency. As part of this transition, the Department undertook a statistical study to bridge the results the 2012 MEPA and 2013 ACCESS for ELLs tests. The study had three purposes:
to prepare concordance tables linking the 2013 ACCESS scores to comparable 2012 MEPA scores;
to provide the basis for determining the attainment of English proficiency in each district and in the state; and
to provide the basis for determining the progress made by Massachusetts English language learners (ELLs) in learning English from 2012 to 2013.
In a related effort, the Department calculated transitional Student Growth Percentiles (SGPs) and provided these to districts in June 2013. The transitional SGPs were calculated from students’ prior MEPA scores and the current year’s ACCESS for ELLs scores. While the SGPs were not used to make progress determinations for students, they were used to assist in determining the relative progress of individual students in learning English, and to establish the validity of the bridging method eventually used to determine progress.
This document provides an overview of the bridge study. A technical report that provides detailed information on the methods used and other considerations will be available in fall 2013.
BackgroundELLs are required to participate in annual statewide assessments of English language proficiency to determine their current levels of performance in English and whether they still require language services as part of their publicly funded instruction. States are required under Title III of the No Child Left Behind Act (2001) to report on the achievement of these students and on their progress in learning English. Progress is determined by comparing the current and prior years’ English proficiency assessment results for each student who has a score in two successive years.
The state’s transition from MEPA in 2012 to the ACCESS for ELLs assessment in 2013 presented a challenge for Massachusetts in meeting the requirement to determine and report the progress of ELLs in learning English. In response, the state conducted a bridge study to link the results of the 2013 ACCESS for ELLs tests and the 2012 MEPA tests, and to estimate the equivalent scores on each test. The study results will be used to determine whether Title III districts and the state have met their Annual Measurable Achievement Objective in 2013 for making progress in learning English (also known as AMAO 1). Local educators and policy makers will also use this information to make reasonable comparisons between scores from the 2012 and 2013 assessments and to make appropriate decisions about whether to provide support and language services to individual ELLs in the 20132014 school year.
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 4
Equipercentile Linking Method Used to Bridge ACCESS for ELLs and MEPA ResultsThe bridge study involved the use of a statistical method called “equipercentile linking,” which bridges the scores from one test to another using percentile ranks. The percentile rank of a score indicates the percentage of scores at that grade level that were lower than the score in question. (For example, if a score of 450 on the grade 5 MEPA test received a percentile rank of 40, then 40 percent of scores at that grade level were lower than 450.)
For each test (2012 MEPA and 2013 ACCESS), the Department assigned a percentile rank to each score at every grade level. The Department then used these percentile ranks to identify a “comparable” MEPA score for each 2013 ACCESS composite score at each grade. For example, at grade 2, an ACCESS score of 291 was considered comparable to a MEPA score of 477 because both scores fell at the 15th percentile. A grade 2 student who scored 291 on the 2013 ACCESS test was therefore assigned an estimated MEPA score of 477 for the purpose of the bridge study.
Table 1 shows a sample of comparable MEPA and ACCESS composite scores for grade 2, based on percentile ranks. The scatterplot in Graph 1 shows the same data in graphical form.
The concordance tables in the Appendix of this report list the comparable scores for ACCESS for ELLs and MEPA side-by-side according to the percentile rank of each score in each grade. The concordance tables were used as the basis for determining progress as described on page 6 of this report.
Table 1Sample Concordance Table:
Estimated Comparable MEPA and ACCESS for ELLs Composite Scores for Grade 2
Concordance Table Based on Equipercentile Linking 2012 MEPA
to 2013 ACCESS for ELLs for Grade 2
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 5
Graph 1Relationship between 2012 MEPA and 2013 ACCESS for ELLs Scores
Assumptions for Conducting the Bridge StudyIn conducting the bridge study, the Department relied on the following key assumptions about the MEPA and ACCESS for ELLs tests, and about the student populations that took them:
The ELL populations in 2012 and 2013 (the two years of the study) were similar demographically; if small differences in the student populations existed, corrections would be made by “weighting” each population statistically.
The ELL subgroup would perform similarly in 2012 and 2013 on the two different assessments of English language proficiency (i.e., MEPA and ACCESS for ELLs), and the subgroup’s proficiency levels would be accurately represented in the results of each test.
Although there are differences between the two tests, each is intended to assess similar content (that is, the English proficiency of ELLs in the areas of speaking, listening, reading, and writing) and ultimately to determine the readiness of students to access standards-based content in English.
However, because the two tests measured different English language development standards, were given in different grade clusters, and used different formats for the oral (listening and speaking) portions, it is important to emphasize that the bridge study results should be regarded as well-informed estimates of the comparable scores on the two tests.
Description of the Student SampleThe bridge study was conducted, and the concordance tables developed, based on two independent samples of assessment results from: a) the students who took the 2012 MEPA test, and b) the students who took the 2013 ACCESS for ELLs test. Students were included in the study only if they took all four subtests: speaking, listening, reading, and writing.
For the purpose of the study, the student sample for each test was divided into two groups: “paired” and “unpaired.” Students in the paired group are those who took all four subtests of both the 2012
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 6
MEPA and the 2013 ACCESS for ELLs tests (that is, the students with paired composite scores). “Unpaired” students are those who took all four subtests of one test, but not of both tests. Table 2 shows the number of students in each group, by grade level. The “paired” and “unpaired” samples were used together to calculate the equipercentile ranks for the concordance tables. Only the “paired” sample was used to make progress determinations.
Table 2 Number of “Paired” and “Unpaired” Test Participants
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 7
Table 3 presents the demographic information for the two student populations (2012 MEPA test-takers and 2013 ACCESS for ELLs test-takers). The percentage of students in each demographic category across both tests is quite similar, and therefore no “weighting” of either population was necessary to correct for differences. Effect sizes near zero, shown in the right column, indicate no or very small group differences. The effect size statistics were calculated using the pooled standard deviations shown in the fourth column of the table.
Table 3Statistical Differences in the Demographics of the Two Student Populations
Other Background VariablesFree/Red. Lunch 80.4% 81.5% .39 -.03Immigrant 26.4% 27.4% .44 -.02Gender (Female) 46.9% 46.8% .50 .00Special Education 14.6% 13.7% .47 .02Attendance Rate 94.9% 94.9% .07 .00Years in MA 3.43 3.44 2.89 .00*Pooled Standard Deviation
Demographic Differences Between Populations taking 2012 MEPA and 2013 ACCESS for ELLs:
All Grades
Annual Measurable Achievement Objectives States are required to report on both the attainment and progress of ELL students in learning English, and whether Title III districts and the state have met their Annual Measurable Achievement Objectives (AMAOs) for attainment and progress, per Title III of the No Child Left Behind law. The bridge study was intended to produce results that could be used by districts and the state in reporting both AMAO 1 (student progress in learning English) and AMAO 2 (attainment of English proficiency). Additional information on AMAOs is available at www.doe.mass.edu/ell/amao/2012/guide.pdf.
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 8
AMAO: Attainment of English ProficiencyThrough 2012, attainment targets for students were based on the MEPA performance levels (Levels 1, 2, 3, 4, and 5). The upper half of MEPA Level 4 (called Level 4 High) signaled the point at which a student could generally become a candidate to exit ELL status. Therefore, the mid-point of Level 4 was chosen as the minimum attainment target. Because the actual scaled score needed to reach Level 4 High varied by grade-span test, the mid-score-point of Level 4 also varied for each grade-span test.
Analysis of the 2012 MEPA data indicated that the characteristics of the ELL student population vary widely from district to district. Therefore, the Department used statewide data to set individual attainment targets for each district based on the number of years its ELL students had been enrolled in a Massachusetts school. The minimum statewide attainment target had been set at one percentage point below the average number of students in the state who scored at the mid-point of Level 4, weighing the number of years (0 through 5+) in which ELL students had been enrolled in Massachusetts schools.
Attainment targets for 2013 were calculated for Massachusetts districts based on this procedure, using the concordance tables to pair each 2013 ACCESS for ELLs test score with its comparable 2012 MEPA score. Attainment results for the state and for each district will be calculated and reported in fall 2013. In 2014, new rules will be proposed, based on guidelines provided to states by the WIDA consortium for gauging attainment of English proficiency.
AMAO: Progress in Learning EnglishThe “Step Approach”
From 2008 to 2012, Massachusetts used the Step Approach shown in Table 4 as the basis for making progress determinations using the MEPA results. The Step Approach entailed dividing the five MEPA performance levels into “steps” as shown in the table. Levels 1 through 4 were divided into two steps each (e.g., Level 2 Low, Level 2 High). Level 5 was divided into five smaller steps. A student was deemed to have “made progress” if the test results from two successive years improved by the requisite number of “steps,” as indicated by the Step Approach Decision Rules (shown adjacent to the table).
For this bridge study, Massachusetts elected to use the same Step Approach method and decision rules that had been previously approved by the U.S. Department of Education (USED) for making progress calculations for AMAO 1. In the study, progress calculations were based on two scores: 1) the student’s actual 2012 MEPA score, and 2) the estimated MEPA score the student would have received in 2013, based on equipercentile linking of his or her ACCESS for ELLs score. After translating these two scores into MEPA performance levels, Massachusetts determined, based on existing Step Approach Decision Rules, whether each student made progress along the continuum of MEPA performance levels from 2012 to 2013. The concordance tables presented in the Appendix of this document show how students’ 2013 ACCESS for ELLs scores were translated into MEPA performance levels.
In 2014, the Department will have two consecutive years of ACCESS for ELLs test results on which to base progress determinations, and will propose new rules to USED for determining student progress along the six-level continuum of ACCESS for ELLs, in consultation with WIDA technical staff.
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 9
Levels Steps
5
13
12
11
10
9
4 8
7
3 6
5
2 4
3
1 2
1
Table 4 The Step Approach Used from 2008 to 2012 to Determine Whether a Student
Made Progress in Learning English, Based on the MEPA Performance Levels
MEPA Performance Levels and Steps MEPA Step Approach Decision Rules
Using the Step Approach and decision rules described above, 61 percent of ELLs in the state made progress learning English from 2012 to 2013, using the scores linked through equipercentile ranking. Table 5 shows statewide progress by grade level.
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 10
For students who took the same grade-span test in successive years (e.g., grade span 5–6 in both years)
Year 1 MEPA score Needed to make progress in Year 2
Level 1 Low to Level 3 Low At Steps 1-5, must advance 2 stepsLevel 3 High to Level 4 High At Steps 6-8, must advance 1 stepLevel 5 At Steps 9-12, must advance 1 step
At Step 13, must maintain step
For students who took a different grade-span test in successive years (e.g., grade span 3–4 in Year 1 and
grade span 5–6 in Year 2)
Year 1 MEPA score Needed to make progress in Year 2
Level 1 Low to Level 4 High At Steps 1-8, must advance 1 stepLevel 5 At Steps 9-13, must maintain step
Table 5Results of 2013 Statewide Progress Determinations by Grade,
Based on Equipercentile Ranking of 2012 MEPA and 2013 ACCESS for ELLs Scoresand the MEPA Step Approach and Decision Rules
Appendix: ACCESS and MEPA Concordance Tables As a result of the bridge study, concordance tables were generated that provide a comparison of ACCESS for ELLs and MEPA composite performance levels and scaled scores. These tables are presented on the following pages. A separate table is provided for each grade, since the relationship between the two sets of scores varies by grade. The tables are intended for use by local educators to identify similar MEPA and ACCESS for ELLs scores for students in each grade level, especially for those students who took both tests.
Massachusetts Department of Elementary and Secondary Education Bridge Study Linking the Results of 2012 MEPA and 2013 ACCESS for ELLs 11