Inter-Rater and Test-Retest Inter-Rater and Test-Retest Reliability of the Early Reliability of the Early Language and Literacy Classroom Language and Literacy Classroom Observation (ELLCO) Observation (ELLCO) Addie Lafferty Addie Lafferty Khawla Obeidat Khawla Obeidat Shelley Gray Shelley Gray Jeanne Wilcox Jeanne Wilcox Arizona State University Tempe, Arizona
13
Embed
Addie Lafferty Khawla Obeidat Shelley Gray Jeanne Wilcox Arizona State University Tempe, Arizona
Inter-Rater and Test-Retest Reliability of the Early Language and Literacy Classroom Observation (ELLCO). Addie Lafferty Khawla Obeidat Shelley Gray Jeanne Wilcox Arizona State University Tempe, Arizona. What is the ELLCO?. A set of three interdependent research tools: - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Inter-Rater and Test-Retest Reliability of Inter-Rater and Test-Retest Reliability of the Early Language and Literacy the Early Language and Literacy Classroom Observation (ELLCO)Classroom Observation (ELLCO)
Purpose: Assess inter-rater and test-retest reliability
Procedure: Administered by two certified speech-language
pathologists at the same timeOne was a bilingual Spanish-English speakerOne was a monolingual English speaker
Two to three week interval between first and second administrations
Characteristics of ParticipantsCharacteristics of Participants
Participants:8 Head Start classrooms
Three classes of 3- to 4-year-oldsFive classes of 4- to 5-year-olds
135 children 115 children identified as English Language
LearnersTeachers & children spoke Spanish and English
throughout the four hour day Teacher demographics: 3 Latino, 3 White, 1
African American, 1 American Indian
ResultsResults
Inter-rater reliability
Classroom Observation & Teacher Interview
Literacy Environment Checklist
Literacy Activities Rating Scale
Time 1 87%
Time 2 92%
Test-retest coefficient
.85 .74 .57
DiscussionDiscussion
The ELLCO had good inter-rater reliability.
The Classroom Observation & Teacher Interview showed good test-retest reliability.
The Literacy Environment Checklist showed adequate test-retest reliability. Scores were affected by the presence or absence of classroom materials from Time 1 to Time 2
The Literacy Activities Rating Scale showed poor test-retest reliability. Scores were affected by the presence or absence of book reading and writing activities from Time 1 to Time 2.
Test-Re-test Reliability Concerns Test-Re-test Reliability Concerns with the with the Literacy Activities Literacy Activities
Rating ScaleRating Scale
The observations at test and re-test were scheduled at the same time of the day and when teachers reported that book reading and writing activities were scheduled to take place.
However, the reliability coefficient was affected by scores on the presence of full-group book reading, number of minutes spent on book-reading, observing an adult model writing, and observing an adult help a child write.
The observation of these behaviors at one time point and not another was the basis of the poor test-retest reliability found here.
Considerations When Using Considerations When Using the ELLCO in Researchthe ELLCO in Research
Consider performing a minimum of two pre-intervention and two post-intervention observations to determine if the scores on the ELLCO are stable and truly representative of the classroom.
Predetermine an amount of time to spend on observations to allow for consistent administration.
When measuring change in a classroom, schedule the observations at the same time as the first observation and when the teacher reports book reading and writing activities will take place to maximize the possibility of observing assessed behaviors.
Suggestions for Future ResearchSuggestions for Future Research
Determination of more controlled directions for performing ELLCO observation.
Consideration of how many observations in a given classroom are needed before stability of scores is obtained.