Findings from the Parent Aware Validation Study...Presentation at the BUILD QRIS National Meeting . Kathryn Tout, Jennifer Cleveland, Winnie Li, Meg Soli & Rebecca Starr . 2 . Parent

Findings from the Parent Aware Validation Study

July 12, 2016

Presentation at the BUILD QRIS National Meeting

Kathryn Tout, Jennifer Cleveland, Winnie Li, Meg Soli & Rebecca Starr

2 Parent Aware Evaluation July 2016

Evaluation Team, Partners and Funders • Jennifer Cleveland • Weilin Li • Meg Soli • Becca Starr • Theresa Sexton • Erin Bultinck • Megan Treinen • Claire Lowe • Team of 15 research

assistants • Kathryn Tout

Partners • Center for Early Education and

Development, University of MN • Wilder Research • SRI International

Evaluation Support • MN Department of Human Services • MN Department of Education • Child Care Aware of Minnesota Funders • Parent Aware for School Readiness • Greater Twin Cities United Way • RTT-ELC grant


Parent Aware Key Features

• Parent Aware is Minnesota’s QRIS • Parent Aware has two rating pathways: Full Rating – Star One through Star Four

• Hybrid system: Stars One and Two – Blocks; Stars Three and Four –

Points • CLASS is used to determine Three or Four Stars for

preschool, center-based programs only • No observation tool is used at any level for family child

care programs Accelerated Rating – Star Four only • Open to programs that are accredited (FCC and centers),

Head Start and school-based pre-kindergarten programs


Parent Aware – Key Features • Parent Aware became

statewide in 2015; a gradual rollout to counties occurred from 2012-2015

• Density of participation in Parent Aware is nearly 90% among programs in the Accelerated pathway; among programs eligible for the Full Rating, participation is low (approximately 10% statewide)


Parent Aware – Key Features

• Star Four is the most common rating in Parent Aware overall (58% of programs)

• Stars One and Two are the most common ratings among programs with Full Ratings

APR Four-Star, 51%

One-Star, 18%

Two-Star, 17%

Three-Star, 7%

Four-Star, 7%

N=2,682 Source: Develop, MN’s Quality Improvement and

Registry Tool, January 2016


What is the Parent Aware Validation Study?

• The Parent Aware validation study examines the extent to which quality ratings are fair, accurate, and meaningful.

• Findings from the Parent Aware validation study

can be used to refine the rating tool and process.


Parent Aware Validation Questions • What is the observed quality of programs with

high and low Parent Aware ratings?

• What developmental gains are children making from fall to spring in Parent Aware-rated programs? • Do gains relate to a program’s Parent Aware

rating? • Do gains relate to observed quality?

• How well does the Accelerated Pathway to

Rating process work to identify high quality? • What are the implications of the findings?


What is the observed quality of programs with high and low Parent Aware ratings?


Measures of Quality • Global quality (ECERS-R, FCCERS-R)

• Conducted in center-based classrooms (ECERS-R) and family child care programs (FCCERS-R).

• Curriculum-related practices (ECERS-E) • Conducted in centers and family child care programs. • Selected subscales measure (1) literacy and (2) math

practices and (3) the extent to which teaching practices are individualized based on children’s unique needs.

• Teacher-child interaction (CLASS) • Conducted in center-based preschool classrooms in

fully-rated centers, accredited centers, Head Start programs and school-based prekindergarten programs.


Number of Programs Included in the Study

One-Star Two-Star Three-Star Four-Star Total

Fully-Rated

30 64 35 61 190

APR n/a n/a n/a 135 135

Total 30 64 35 196 325


Does observed global quality in center-based programs differ by rating level?

Yes. Higher-rated center-based programs (Three- and Four-Star) have higher total ECERS-R scores than lower-rated programs (One- and Two-Star).

3.69 4.03

0

1

2

3

4

5

6

7

Star 1, 2 (n = 35) Star 3, 4 (n = 111)

ECERS-R

*


Does the quality of teacher-child interactions differ in center-based programs at different rating levels?

No. CLASS dimensions look similar in Three-and Four- Star programs and in One- and Two-Star programs.

6.08 6.23

2.43

6.09 6.27

2.48

0

1

2

3

4

5

6

7

CLASS - ClassroomOrganization

CLASS - Emotional Support CLASS - Instructional Support

Star 1, 2 (n = 59) Star 3, 4 (n = 202)


Does the quality of curriculum practices differ in center-based programs at different rating levels?

Yes. Higher-rated center-based programs score higher on language and math practices and on a measure of individualized planning based on children’s needs compared to lower-rated programs.

3.68

2.71

1.09

4.2

3.24

1.55

0

1

2

3

4

5

6

7

ECERS-E Language ECERS-E Math ECERS-E PlanningStar 1, 2 (n = 35) Star 3, 4 (n = 110)

*

*

*


Does the observed quality of family child care programs differ at different rating levels?

No significant differences were found between lower- and higher–rated programs on global quality (FCCERS-R).

3.32 3.4

0

1

2

3

4

5

6

7

Star 1,2 (n = 31) Star 3,4 (n = 24)

FCCERS-R


Does the quality of curriculum practices differ in family child care programs at different rating levels?

No significant differences were found between lower- and higher-rated family child care programs on language and math practices or on a measure of individualized planning based on children’s needs.

3.19 2.65

1.13

3.1 2.57

1.42

0

1

2

3

4

5

6

7

ECERS-E Language ECERS-E Math ECERS-E Planning

Star 1, 2 (n = 31) Star 3, 4 (n = 24)


Parent Aware Validation Findings Summary

Center-based programs: On four of seven measures, observed quality was higher in programs with higher ratings. Family child care programs: Observed quality was not significantly different in programs with different ratings.


What developmental gains are children making from fall to spring in Parent Aware-rated programs?


Who are the 1181 children in the Parent Aware evaluation? • 4 years old, in their year

before Kindergarten

• 51% male • 64% white, 15% African

American or African, 4% Asian, 4% Hispanic, 8% Other, 5% missing

• 51% from greater MN, 44% from the Metro, 5% missing

• 62% low-income Defined as ≤ 185% of the federal poverty level Low-income children were prioritized for recruitment

• 69% excellent English skills, 24% good English skills*

• 35% received Scholarships*

*indicates variables with missing data for a portion of the sample


Measures of Child Development

• Language and literacy skills • Phonological

awareness • Print knowledge • Expressive

vocabulary • Early math problem

solving • Executive function

• Approaches to learning • Persistence

• Social emotional skills • Social competence • Anger/aggression • Anxiety/withdrawal

• Basic concepts (color, size, number)

• Body mass index (weight risk assessment)


Do children’s developmental gains relate to their program’s Parent Aware rating or observed quality?


What statistical approaches are used to examine how Star ratings are associated with children’s development? • Many child- and parent-level characteristics are related to

children’s developmental skills • Rigorous statistical models are used to account for these

demographic characteristics and the fact that our sample of children is “nested” within ECE programs.

• Control variables include: • Child gender • Child race / ethnicity • Child English language levels • Family low-income status • Child attendance

• Parental education • Fall assessment scores • Dosage of ECE program

experience


For each developmental skill, we examine:

• Whether there are significant differences in fall to spring gains that are associated with the Parent Aware rating of the child’s program (low, high)

• Whether the association between program rating and gains is

stronger for low income children • Whether measures of observed quality are related to fall to

spring gains


Do children make larger gains on school readiness skills when they are in programs with high ratings?

• Children who attend Three- and Four-Star rated programs made larger gains on teacher ratings of persistence (which measures skills like paying attention, determination, cooperation and concentration) than did children who attend One- and Two-Star rated programs.

• Low-income children who attend programs with high ratings made larger gains on teacher ratings of social competence which measure things like compromising, sharing, and working well in groups.

• Low-income children who attend programs with high ratings made larger gains on a measure of print knowledge.


Do children make larger gains on school readiness skills when they are in programs with high ratings?

Language and Literacy Skills:

1. Phonological Awareness 2. Print Knowledge 3. Expressive Vocabulary

Social and Emotional Skills:

4. Social Competence 5. Anxiety/Withdrawal 6. Anger/Aggression

Approaches to Learning:

9. Persistence

Early Math Skills:

7. Problem Solving

Cognitive Skills 8. Executive Function

GREEN indicate skills with greater gains when children were in high rated programs

Yes. Higher Parent Aware ratings are associated with gains in children’s skills in three of five developmental domains, particularly for low-income children.


Does observed quality relate to children’s development? • Global quality in center-based programs (as

measured by ECERS-R) is linked to children’s gains on a measure of print knowledge and phonological awareness*.

• Language practices (ECERS-E) are linked to children’s gains in expressive vocabulary and social competence.

• Instructional Support in center-based programs (CLASS) is linked to children’s gains on executive function (working memory and inhibitory control).

*Found in low income sample only



Associations in the expected direction between ratings and children’s development were found on three of nine outcomes (persistence, social competence and print knowledge). Some measures of observed quality were related to children’s development. Higher rated programs were documented to provide more of these practices than lower-rated programs.


How well does the Accelerated Pathway to Rating process work to identify high quality?


Does observed global quality in center-based programs differ by rating pathway? No. APR programs look similar to fully-rated Three- and Four-Star

programs. APR programs and fully-rated Three-and Four-Star programs have higher total ECERS-R scores than lower-rated programs (One- and Two-Star).

3.69 4.05 4.02

0

1

2

3

4

5

6

7

ECERS-R

Star 1, 2 (n = 35) Star 3, 4 (n = 18) APR (n = 93)

*

*


Does the quality of teacher-child interactions differ in center-based programs by rating pathway?

Yes. CLASS Instructional Support is higher in fully-rated Three- and Four-Star programs than in APR programs and One- and Two-Star programs.

6.08 6.23

2.43

6.17 6.34

2.73

6.04 6.23

2.35

0

1

2

3

4

5

6

7

CLASS - Classroom Organization CLASS - Emotional Support CLASS - Instructional Support

Star 1, 2 (n = 59) Star 3, 4 (n = 72) APR (n = 130)

* *


Does the quality of curriculum practices differ in center-based programs by rating pathway?

Yes. APR programs score higher than fully-rated Three- and Four-Star and One- and Two-Star rated programs on language and math practices. Fully-rated Three and Four-Star & APR programs score higher than One- and Two-Star rated program on individualized planning.

3.45

2.68

1.11

3.43

2.82

1.5

4.23

3.23

1.53

0

1

2

3

4

5

6

7

ECERS-E Language ECERS-E Math ECERS-E Planning

Star 1, 2 (n = 66) Star 3, 4 (n = 42) APR (n = 94)

*

*

*

*

*

*



Differences by rating pathway were observed. On balance however, the differences are not systematic . Some findings favor fully-rated programs and others favor APR programs. CLASS Instructional Support scores are higher in Three- and Four-Star rated programs than in other fully-rated and APR programs.


Key Validation Findings • The findings on observed quality and children’s development

provide positive, initial support for the validity of the Parent Aware ratings in supporting meaningful quality differences that are related to children’s development in expected ways.

• Overall, the Accelerated Pathway to Rating appears to function effectively to identify programs that engage in practices to support school readiness, particularly for low-income children. APR programs and Three- and Four-Star fully-rated programs both have strengths, according to the observational data and findings on children’s development.

• Further research is needed to identify needs and options for

strengthening the rating process for family child care programs.


Key Implications • Across all quality levels, program types, and rating

pathways, programs in Parent Aware, including those that have achieved a Three- or Four-Star full-rating and APR programs, could benefit from quality improvement efforts.

• CLASS coaching appears to be supporting higher scores on Instructional Support among fully-rated Three- and Four-Star centers.

• The findings do not indicate that APR programs would be differentiated more successfully by requiring a full-rating process with the current set of Parent Aware indicators.


Key messages when sharing the findings with stakeholders

• Parent Aware is still early in statewide implementation.

• Even though quality is differentiated by the ratings: • The magnitude of the

differences is small. • The overall quality – across

rating levels and program types – is lower than expected.

• Major changes to the rating system are not warranted by the findings.

• Proposed changes should be approached cautiously and with a field test if possible.


Lessons learned about conducting validation studies

• Engage stakeholders early and often about the purpose of the study

• Use multiple, diverse measures of observed quality if possible

• Develop creative, individualized recruitment strategies

• Connect with other research initiatives


Thank you!

Contact Kathryn Tout [email protected]

with questions or comments

mailto:[email protected]

Findings from the Parent Aware Validation Study...Presentation at the BUILD QRIS National Meeting . Kathryn Tout, Jennifer Cleveland, Winnie Li, Meg Soli & Rebecca Starr . 2 . Parent

Documents