Evaluation of the Michigan Literacy Progress Profile … of the Michigan Literacy Progress Profile (MLPP) ... Gina Cervetti Robert Carpenter Sharman Siebenthal Alison Paris Brigette

Evaluation of the MichiganLiteracy Progress Profile

(MLPP)

Robert D. CarpenterUniversity of Michigan

Center for the Improvement of Early Reading Achievement (CIERA)

Contact information:e-mail: [email protected]

CIERA: www.ciera.org

Evaluation TeamMichigan State University University of Michigan

Dr. P. David Pearson Dr. Scott Paris, ProfessorDistinguished Professor of Education CIERA, Department of Psychology,

Combined Program inEducation and Psychology

Gina Cervetti Robert CarpenterSharman Siebenthal Alison ParisBrigette Laier Graduate ResearchersGraduate Researchers CIERA, Combined Program in

Education and Psychology

Denise LarabellResearch Assistant

Ingham Intermediate School District

Tamara Bashore-Berg, SupervisorDepartment of Planning and Evaluation

Kathy Humphrey, Evaluation SpecialistDepartment of Planning and Evaluation

Jonathan Flukes, Evaluation AssistantDepartment of Planning and Evaluation

Kelly Trout, Evaluation AssistantDepartment of Planning and Evaluation

Allen Strozewski, Computer TechnicianDepartment of Planning and Evaluation

Chris Quinn, SecretaryDepartment of Planning and Evaluation

Overview

• Overall Evaluation

• Year 1 Test/Retest Reliability

• Year 1 Concurrent Validity

• Year 1 Summary

• Year 2 Overview

Overall Evaluation

Primary Goal:

To evaluate the reliability, validity,and utility of the MLPP

Evaluation Timeline

–Year 1•Test/Retest Reliability•Concurrent Validity

–Year 2•Predictive Validity•Consequential Validity

Purposes of Year 1

• To collect baseline data for long-termpredictive validity of the MLPP

• Examine the test/retest reliability of theMLPP enabling skills

• Examine the test/retest reliability of theMLPP oral reading assessments

• To test the concurrent validity of both theenabling and oral reading tasks

Method

• Data were collected in the spring andfall of 2001

• For test/retest reliability studentswere tested on the same tasks 2-4weeks apart by teachers andresearchers

• Concurrent validity established usingteacher and researcher MLPP dataand researcher collected TPRI andGates data

Sample

• Four sites across the state wereselected because they representedrural, urban, and suburbancommunities and were strongimplementors of the MLPP

• More than 700 K-3 studentsapproximately 25% in each grade

• 41 % Female

• 68 % Caucasian

UM Specific Procedures

• Met administrators and teachers toexplain project in spring 2001

• Retested students using the sameMLPP assessments and proceduresas teachers 2-4 weeks after theteacher administration (spring 2001)

• Tested students using the TPRI(spring 2001) and Gates (fall 2001) atMilan during a similar 4-week window

MLPP EnablingComponents

• Letter Identification

• Letter-Sound Identification

• Hearing and Recording Sounds

• Phonological Awareness–Rhyme–Onset Rim and Blending–Segmentation

• Concepts of Print

• Known Words

MLPP Oral ReadingFramework

• Sight Word/Decodable Word List

• Oral Reading–Accuracy–Fluency–Retelling–Comprehension

Test/Retest Correlationsfor MLPP Enabling TasksTask Milan Willow

RunLetter ID .94*** .97***

Letter Sound ID .91*** .74***

Phonemic Awareness

Rhyming .47* .73***

Blending .78*** .91***

Segmentation .10 .68***

H and R Sounds .93***

Concepts of Print .65*** .33

Known Words .73***

* = p< .05; ** = p< .01; *** = p< .001

Letter IdentificationMilan Scatterplot of Teacher and Researcher Scores

Researcher administration

5550454035302520151050

Tea

cher

Adm

inis

trat

ion

55

50

45

40

35

30

25

20

15

10

5

0

Grade

First Grade

Kindergarten

Willow Run Scatterplot of Teacher and Researcher Scores

Researcher Administration

5550454035302520151050

Tea

cher

Ad

min

istr

atio

n

55

50

45

40

35

30

25

20

15

10

5

0

Overall Regression Line

Agreement Line

Regression LineAgreement Line

Letter Sound IDMilan Scatterplot of Teacher and Researcher Scores


302520151050-5

Teac

her A

dmin

istra

tion

30

25

20

15

10

5

0

-5

Grade

First Grade

Kindergarten



302520151050-5

Tea

cher

Adm

inis

trat

ion

30

25

20

15

10

5

0

-5

Overall Regression LineAgreement Line

Regression Line

Agreement Line

Phonemic Awareness- RhymingMilan Scatterplot of Teacher and Researcher Scores First Grade Only


181614121086420

Tea

cher

Adm

inis

trat

ion

18

16

14

12

10

8

6

4

2

0



181614121086420

Teac

her A

dmin

istra

tion

18

16

14

12

10

8

6

4

2

0

Grade

First Grade

KindergartenOverall Regression Line

Agreement Line

Regression Line

Agreement Line

Phonemic Awareness- BlendingMilan Scatterplot of Teacher and Researcher Scores


181614121086420

Tea

cher

Ad

min

istr

atio

n

18

16

14

12

10

8

6

4

2

0



181614121086420

Tea

cher

Adm

inis

trat

ion

18

16

14

12

10

8

6

4

2

0

Regression Line

Agreement Line


Phonemic Awareness- SegmentationMilan Scatterplot of Teacher and Researcher Scores


1086420

Tea

cher

Ad

min

istr

atio

n

26

24

22

20

18

16

14

12

10

8

6

4

2

0



1086420-2

Tea

cher

Ad

min

istr

atio

n

10

8

6

4

2

0

-2

Regression Line

Agreement Line


Hearing and Recording SoundsWillow Run Scatterplot of Teacher and Researcher Scores


4035302520151050

Tea

cher

Ad

min

istr

atio

n

40

35

30

25

20

15

10

5

0Regression LineAgreement Line

Concepts of Print Milan Scatterplot of Teacher and Researcher Scores


242220181614121086420

Tea

cher

Adm

inis

trat

ion

24

22

20

18

16

14

12

10

8

6

4

2

0

Grade

First Grade

Kindergarten



242220181614121086420

Tea

cher

Ad

min

istr

atio

n

24

22

20

18

16

14

12

10

8

6

4

2

0


Regression Line

Agreement Line

Known WordsWillow Run Scatterplot of Teacher and Researcher Scores


65605550454035302520151050

Tea

cher

Ad

min

istr

atio

n

65

60

55

50

45

40

35

30

25

20

15

10

5

0

Regression Line

Agreement Line

Test/Retest Correlationsfor MLPP Oral Reading

Task Milan WillowRun

Word Recognition(Highest List)

.66*** .78***

Word Recognition(Scale)

.94*** .98***

Accuracy .51*** .46***

Fluency .54**

Rate .78***

Comprehension .35*

Retelling .53*** .10

* = p< .05; ** = p< .01; *** = p< .001

Highest Sight Word ListMilan Scatterplot of Teacher and Researcher Scores


211815129630

Teac

her A

dmin

istra

tion

21

18

15

12

9

6

3

0

Grade

Third Grade

Second Grade



211815129630

Teac

her A

dmin

istra

tion

21

18

15

12

9

6

3

0

Grade

Third Grade

Second Grade

First GradeOverall Regression Line

Agreement Line


Agreement Line

Total Sight Word ScoreFigure 19. Milan Scatterplot of Teacher and Researcher Scores


180160140120100806040200

Teac

her A

dmin

istra

tion

180

160

140

120

100

80

60

40

20

0

Grade

Third Grade

Second Grade

Figure 20. Willow Run Scatterplot of Teacher and Researcher Scores


120100806040200

Teac

her A

dmin

istra

tion

120

100

80

60

40

20

0

Grade

Third Grade

Second Grade

First Grade


Agreement Line


Agreement Line

AccuracyMilan Scatterplot of Teacher and Researcher Scores


1101009080706050

Teac

her A

dmin

istra

tion

110

100

90

80

70

60

50

Grade

Third Grade

Second Grade

Willow Run Scatterplot of Teacher and Researcher Scores in Decimals


1.101.00.90.80.70.60.50

Teac

her A

dmin

istra

tion

1.10

1.00

.90

.80

.70

.60

.50

Grade

Third Grade

Second Grade

First GradeOverall Regression Line

Agreement Line


Fluency

Highest Passage Scatterplot of Teacher and Researcher Scores


543210

Teac

her A

dmin

istra

tion

5

4

3

2

1

0

Grade

Third Grade

Second GradeOverall Regression Line

Agreement Line

Reading Rate

Scatterplot of Highest Passage Reading Times


350300250200150100500

Teac

her A

dmin

istra

tion

350

300

250

200

150

100

50

0

Grade

Third Grade


Agreement Line

Comprehension

Scatterplot for Highest Passage


1109070503010-10

Teac

her A

dmin

istra

tion

110

90

70

50

30

10

-10

Grade

Third Grade


Agreement Line

RetellingMilan Scatterplot of Highest Passage Percent Propositions


80706050403020100

Teac

her A

dmin

istra

tion

80

70

60

50

40

30

20

10

0

Grade

Third Grade

Second Grade

Willow Run Scatterplot of Retelling Rubric Scores


4.54.03.53.02.52.01.51.0.5

Teac

her A

dmin

istra

tion

4.5

4.0

3.5

3.0

2.5

2.0

1.5

1.0

.5

Grade

Third Grade

Second Grade

First Grade


Agreement Line


Concurrent Validityfor MLPP Tasks

Task TPRI GMRT

Letter ID .94*** .82**

Phonemic Awareness .19

Rhyming .43**

Blending .81***

Word Recognition .79***

Accuracy .41*

Comprehension .07 .71***

* = p< .05; ** = p< .01; *** = p< .001

Summary ofYear 1 Findings

• High reliability for enabling skills

• Moderate reliability for oral reading tasks

• Moderate concurrent validity with theTPRI and the GMRT

Year 2

• We are currently collecting MEAP dataon 3rd grade students from year one inaddition to the MEAP data ofparticipants in previous MLPP studies tomeasure predictive validity

• Spring 2002 we will collect surveys andinterviews of students, teachers andadministrators to determine theconsequential validity of the MLPP

Evaluation of the Michigan Literacy Progress Profile … of the Michigan Literacy Progress Profile (MLPP) ... Gina Cervetti Robert Carpenter Sharman Siebenthal Alison Paris Brigette

Documents