Cheri Quinlan NJ Department of Education Linda Forrest Center for Applied Second Language Studies.

Cheri QuinlanCheri QuinlanNJ Department of EducationNJ Department of Education

Linda ForrestLinda ForrestCenter for Applied Second Language Center for Applied Second Language

StudiesStudies

A.A. Background:Background:

• MandateMandate:1996-Present:1996-Present

• Challenge:Challenge: Lack of a reliable and Lack of a reliable and valid system of accountability that valid system of accountability that measures student achievement and measures student achievement and the development of quality programsthe development of quality programs

• Need: Need: student performance data student performance data that shows efficacy of certain that shows efficacy of certain program modelsprogram models

BB. Goal:. Goal:• Universal access to quality world Universal access to quality world

languages programs that lead to higher languages programs that lead to higher levels of student proficiencylevels of student proficiency

CC. . The Approach:The Approach:• Integration of three key componentsIntegration of three key components

– AssessmentAssessment– Professional developmentProfessional development– PolicyPolicy

AssessmentAssessment: : The STAMP TestThe STAMP Test Chinese, French, German, Italian, Chinese, French, German, Italian,

Japanese and Spanish Japanese and Spanish Reading and SpeakingReading and SpeakingTarget: 60,000 Grade 8 studentsTarget: 60,000 Grade 8 students

Professional DevelopmentProfessional DevelopmentOnsite and/or OnlineOnsite and/or OnlineData-driven instruction Data-driven instruction

PolicyPolicyAnticipated policy revisions based on Anticipated policy revisions based on

efficacy of assessment project efficacy of assessment project Project data will be used as baseline for Project data will be used as baseline for

formal assessment formal assessment

20062006 Test Test ##

20072007 Test Test ##

20082008 Test Test ##

ChinesChinesee

FrenchFrench

GermaGermann

JapaneJapanesese

SpanisSpanishh

ItalianItalian

TotalTotal

6363

27172717

496496

1616

12,612,68484

NANA

15,915,97676

ChinesChinese e

FrenchFrench

GermaGermann

JapaneJapanesese

SpanisSpanishh

ItalianItalian

TotalTotal

00

3,3523,352

766766

1010

16,5016,5077

842842

21,4721,4777

ChinesChinesee

FrenchFrench

GermaGermann

JapaneJapanesese

SpanisSpanishh

ItalianItalian

TotalTotal

105105

4949

3,6133,613

694694

1717

15,9115,9144

1,0791,079

21,4721,4711

Number of students placing at Number of students placing at Novice-Novice-HighHigh

or higher for Speaking:or higher for Speaking:

LanguageLanguage % Y1% Y1 %Y2%Y2 %Y3%Y3

ChineseChinese 60%60% NANA 44.44%44.44%

57.45%57.45%

FrenchFrench 26%26% 34%34% 39.79%39.79%

GermanGerman 40.5%40.5% 18%18% 40.12%40.12%

JapaneseJapanese 12.5%12.5% 50%50% 41.18%41.18%

SpanishSpanish 29%29% 29.8%29.8% 35.88%35.88%

ItalianItalian NANA 10%10% 23.8%23.8%

D. Summary DataD. Summary Data::

• Do not reflect anticipated student proficiency Do not reflect anticipated student proficiency levels as a result of K-8 instructionlevels as a result of K-8 instruction

• Indicate that students are consistently Indicate that students are consistently scoring higher on speaking across all scoring higher on speaking across all languages languages

• Is similar to proficiency level distributions for Is similar to proficiency level distributions for 1st year programs across the country where 1st year programs across the country where the majority of students are located in the the majority of students are located in the Novice-Low and Novice-Mid range of Novice-Low and Novice-Mid range of proficiencyproficiency

• CASLS report shows correlation with various CASLS report shows correlation with various program models and student performanceprogram models and student performance

Positive Outcomes:Positive Outcomes: 100% of supervisor respondents and over 100% of supervisor respondents and over

86% of teacher level respondents rated the 86% of teacher level respondents rated the overall experience with STAMP and this overall experience with STAMP and this test cycle as Excellent, Very Good or Good. test cycle as Excellent, Very Good or Good.

Delivered 21,709 tests to 108 districts in Delivered 21,709 tests to 108 districts in all regions of the state all regions of the state

Increased exposure to second language Increased exposure to second language

testing and direct involvement from testing and direct involvement from district and building administration during district and building administration during the test deliverythe test delivery

Fewer technology problems reported- Fewer technology problems reported- most of which were traced to district and most of which were traced to district and building readiness issuesbuilding readiness issues

Positive Outcomes Continued:

Students enjoyed the test and the test environment using technology to showcase their language abilities.

Many districts and teachers expressed enthusiasm about future STAMP testing.

User comments from teachers and supervisors for specific application of STAMP and logistical improvements will informed the improvement of the next round of testing.

Additional InformationAdditional Information::

• This project is the largest data collection This project is the largest data collection event in second languages at the 8th grade event in second languages at the 8th grade level ever undertaken in the United States. level ever undertaken in the United States.

• It is anticipated that the data will be valuable It is anticipated that the data will be valuable to language researchers and program to language researchers and program directors throughout the country as they directors throughout the country as they endeavor to establish highly effective endeavor to establish highly effective language programs. language programs.

• http://www.state.nj.us/njded/aps/cccs/wl/g8assess/

Policy RecommendationsPolicy Recommendations

• Assess all 8th graders annually.Assess all 8th graders annually.

• Provide districts with incentives to Provide districts with incentives to offer exemplary language offer exemplary language programs.programs.

• Provide students with incentives to Provide students with incentives to reach higher levels of proficiency.reach higher levels of proficiency.

2007 High School Random Sample Results2007 High School Random Sample Results

• Level II students - 170 Spanish and 161 FrenchLevel II students - 170 Spanish and 161 French

• Over 60% of students in both languages are Over 60% of students in both languages are scoring at the Novice-High level or higher. This scoring at the Novice-High level or higher. This level is higher than national data at the end of 2 level is higher than national data at the end of 2 years of language study. years of language study.

• Speaking and writing skills are higher than Speaking and writing skills are higher than reading. reading.

• Intermediate range were stronger in writing than Intermediate range were stronger in writing than in the other skills which seems to indicate that in the other skills which seems to indicate that writing is a critical area of focus for these writing is a critical area of focus for these students/courses.students/courses.

Study detailsStudy details• Grade Eight Proficiency Assessment in Grade Eight Proficiency Assessment in

World LanguagesWorld Languages• FLAP grant to NJ DOEFLAP grant to NJ DOE• 21,709 Grade 8 students 21,709 Grade 8 students • 156 schools in 108 districts 156 schools in 108 districts

• Chinese, Spanish, French, German, Chinese, Spanish, French, German, Italian, Japanese Italian, Japanese

• STAMP – delivered by Avant AssessmentSTAMP – delivered by Avant Assessment• Analysis by CASLSAnalysis by CASLS

The STAMP AssessmentThe STAMP Assessment

STAMPSTAMP(Standards-based Measurement

of Proficiency)

STAMPSTAMP(Standards-based Measurement

of Proficiency)

Reading Writing Speaking Listening

Developed by Center for Applied Second Language StudiesAdministered by Avant Assessment (formerly Language Learning Solutions)

Novice

Intermediate

Advanced

Superior

ACTFL

Benchmarks

Higher Level

Benchmarks

Benchmark Level 1

Benchmark Level 2

Benchmark Level 3

Benchmark Level 4

Benchmark Level 5

Benchmark Level 6

NJ State Standard: Novice-

High

STAMP ScoringSTAMP Scoring

• ReadingReading– Semi-adaptive presentationSemi-adaptive presentation– Benchmark Level QualifiedBenchmark Level Qualified

• SpeakingSpeaking– Trained gradersTrained graders– Rubric has 3 componentsRubric has 3 components

• Text typeText type• ComprehensibilityComprehensibility• Task completionTask completion

– Final scores converted to Benchmark levelsFinal scores converted to Benchmark levels

IssuesIssues

• Speaking vs. Reading proficiencySpeaking vs. Reading proficiency• Duration of instructionDuration of instruction• Intensity of instructionIntensity of instruction• SES factorsSES factors• Heritage learnersHeritage learners

Speaking vs. ReadingSpeaking vs. Reading

• Speaking much higher than reading Speaking much higher than reading in 2007.in 2007.– Met state standard:Met state standard:

• ReadingReading 12%12%• SpeakingSpeaking 25%25%

Key Point #1Key Point #1

• Speaking Speaking proficiency is proficiency is much higher much higher than reading than reading proficiency and proficiency and has improved has improved since 2007. since 2007.

Speaking scores almost 3 times higher than reading.Increase in speaking is about 6%.

Duration of instructionDuration of instruction

• Language programs must provide Language programs must provide sufficient hours sufficient hours of instruction in of instruction in order for students to meet Grade 8 order for students to meet Grade 8 state standards.state standards.

How much time is How much time is needed?needed?

• Estimated Estimated lifetimelifetime hours of instruction hours of instruction• ““Instructional Unit”Instructional Unit”

1 IU = 108 hours

3 hours / week for 36 weeks a year

Non-heritage students

0%

10%

20%

30%

40%

50%

60%

1 IU or less 2 - 3 IUs 3 - 4 IUs 5 IUs or more

Perc

ent M

eeti

ng S

tand

ard

Number of Instructional Units (1 IU = 108 hours)

Students Meeting NJ State Standards in Grade 8

2007 Reading

2007 Speaking

ConclusionsConclusions

• Students need Students need sufficientsufficient instructional time instructional time– increasing hours = more students increasing hours = more students

meeting state standardmeeting state standard– students need at least 5 IUs (540 hours) students need at least 5 IUs (540 hours)

for high rates of success for high rates of success • Begin instruction Begin instruction as early as possibleas early as possible

– ideally prior to Grade 6, as with other ideally prior to Grade 6, as with other subjects subjects

– beginning at Grade 6 or later = need to beginning at Grade 6 or later = need to devote significant hours to language in devote significant hours to language in order to reach NH proficiencyorder to reach NH proficiency

Key Point #2Key Point #2

• Speaking proficiency has improved for Speaking proficiency has improved for all students, irrespective of the numberall students, irrespective of the numberof hours of of hours of instruction instruction they have they have received.received.

0%

10%

20%

30%

40%

50%

60%

1 IU or less 2 - 3 IUs 3 - 4 IUs 5 IUs or more

Perc

ent M

eeti

ng S

tand

ard

Number of Instructional Units (1 IU = 108 hours)

Students Meeting NJ State Standards in Grade 8

2007 Reading

2008 Reading

2007 Speaking

2008 Speaking

Intensity of instructionIntensity of instruction

• Language programs which meet Language programs which meet several times several times each week during the each week during the whole school year whole school year are generally are generally more effective than an equivalent more effective than an equivalent number of hours in a partial year number of hours in a partial year program.program.

What type of schedule?What type of schedule?

• sessions / sessions / week week

• weeks / yearweeks / year• Non-heritage Non-heritage

learnerslearners• Students with Students with

more than 3 IUs more than 3 IUs excludedexcluded

• Reading and Reading and speaking show speaking show similar patternssimilar patterns 0%

5%

10%

15%

20%

25%

30%

Partial Yr1-2 D/Wk

Partial Yr3-5 D/Wk

Full Yr1-2 D/Wk

Full Yr3-5 D/Wk

Perc

ent M

eeti

ng S

tate

Sta

ndar

d

Program Type

Students Meeting NJ State Standard for Speaking

2007

ConclusionConclusion• Need sufficient Need sufficient durationduration

– More effective programs > 21 weeks / year More effective programs > 21 weeks / year • Need sufficient Need sufficient intensityintensity

– More effective programs >= 3 times / weekMore effective programs >= 3 times / week• To attain to significant proficiency outcomes, a To attain to significant proficiency outcomes, a

program needs to have program needs to have a minimum of 5,000 a minimum of 5,000 minutesminutes of instruction time spread across the of instruction time spread across the school year. school year.

• Even when the total number of hours per year Even when the total number of hours per year is held constant, students perform better when is held constant, students perform better when instruction time is instruction time is evenly distributed across evenly distributed across the school yearthe school year. .

• Both the Both the distribution of time distribution of time and the and the minimum minimum amount of timeamount of time are significant factors. are significant factors.

Key Point #3Key Point #3 • Speaking proficiency has improved across all Speaking proficiency has improved across all

program models. However programsprogram models. However programs

meeting 3-5 meeting 3-5 times each times each week week during the during the whole whole school year school year are much are much more more effective effective than other than other models.models.

0%

5%

10%

15%

20%

25%

30%

Partial Yr1-2 D/Wk

Partial Yr3-5 D/Wk

Full Yr1-2 D/Wk

Full Yr3-5 D/Wk

Perc

ent M

eeti

ng S

tate

Sta

ndar

d

Program Type

Students Meeting NJ State Standard for Speaking

2008 Change

2007

SES FactorsSES Factors District Factor Groups (DFG)District Factor Groups (DFG) ‘‘Low’, ‘Mid’, ‘High’Low’, ‘Mid’, ‘High’ Total lifetime hours of instructionTotal lifetime hours of instruction

estimated for each student and lifetime estimated for each student and lifetime IUs calculatedIUs calculated

1 IU = 108 hours 1 IU = 108 hours Non-heritage learnersNon-heritage learners

Conclusion - 2007Conclusion - 2007

• Students attending schools in all District Students attending schools in all District Factor Groups have similar speaking Factor Groups have similar speaking proficiency after proficiency after 5 IUs 5 IUs of instruction.of instruction.

• Given equal amounts of instruction Given equal amounts of instruction students in all schools can acquire students in all schools can acquire similar levels of similar levels of speakingspeaking proficiency. proficiency.

• ReadingReading skills pattern with other skills pattern with other academic subjects. Lower SES schools academic subjects. Lower SES schools tend to lag higher SES schools.tend to lag higher SES schools.

Hours of instruction & Hours of instruction & SESSES

• Grade 8 students Grade 8 students in Low DFG in Low DFG schools have schools have received, received, on on averageaverage, only 1/3 , only 1/3 the hours of the hours of instruction as instruction as students in High students in High DFG schools.DFG schools.

0

50

100

150

200

250

300

350

400

Low Mid High

Ho

urs

of

Lan

gu

age

Inst

ruct

ion

DFG Category

Median Lifetime Hours of Foreign Language Instruction of Grade 8

Students by DFG Groupings

Key Point #4Key Point #4 • Students in lower DFG schools made Students in lower DFG schools made

gains in both speaking and reading.gains in both speaking and reading.Higher Higher DFG DFG schools schools made made gains in gains in speaking speaking only.only.

Heritage StudentsHeritage Students

• 2007 2007 – Heritage students are more likely to Heritage students are more likely to

meet state standards.meet state standards.

– Heritage students are similar across all Heritage students are similar across all DFG schools.DFG schools.

Key Point #5Key Point #5

• Number of heritage students Number of heritage students meeting the standard increased for meeting the standard increased for bothboth reading and speaking in 2008. reading and speaking in 2008.

Take-home messageTake-home message

• DurationDuration– teach language teach language every week every week all yearall year

• IntensityIntensity– teach 3 – 5 class sessions per weekteach 3 – 5 class sessions per week

• Lifetime hoursLifetime hours– teach at least 540 hours during Grades teach at least 540 hours during Grades

1-81-8

• Heritage students lead the way to Heritage students lead the way to world language successworld language success

Cheri QuinlanCheri [email protected]

Linda ForrestLinda [email protected]