Jan 23, 2016

Patterns in Child Outcomes Summary Data:. Analytic Approaches and Early Findings from the ENHANCE Project. Cornelia Taylor, Lauren Barton, Donna Spiker September 19-21, 2011. Measuring and Improving Child and Family Outcomes Conference New Orleans, LA. Today’s session. - PowerPoint PPT Presentation

Slide 1September 19-21, 2011

New Orleans, LA

from the ENHANCE Project

Overview of ENHANCE: Research Underway on the Validity of the Child Outcomes Summary Form (COSF)

Are COSF data valid and reliable? This session will provide an overview of a research project underway to investigate the validity of the COSF. Plans for using information learned to provide better guidance about the COSF and implementing the COSF process will be shared. The group will discuss the kinds of validity information needed by States and specific content that might be important to investigate further in the studies

1

Identify the purpose and approach of the state data study

Describe some preliminary findings from initial states involved in the state data study

Explain how other states could examine their own data in the same way as that presented

Discuss any emerging implications for validity of the COS and for interpreting individual state data

Today’s session

Progress of project efforts to investigate the validity of the Child Outcomes Summary process

2

Origins of ENHANCE

COS Process

Implemented > 40 States, Little Systematic Validation for Use in Accountability

Origins of ENHANCE

COS Process

?

Early Evidence

Belief in potential for COS process to be valid based on:

Existing literature: team-based decision-making can be reliable and valid

Existing literature: teams are effective in identifying individual children’s functioning so that they can plan and deliver appropriate services

Early data from states: pilot sites, small n’s showing similarity in distributions, sensible patterns for subgroups

Anecdotal data from trainers: participants reach decisions fairly easily and consistently

ENHANCE

Project launched by the Early Childhood Outcomes Center (ECO) and SRI International

Funded by the U.S. Dept. of Education, Institute for Educational Sciences – July 1, 2009

Series of studies designed to find out:

the conditions under which the Child Outcomes Summary (COS) Process produces meaningful and useful data for accountability and program improvement

the positive and/or negative impact of the COS process on programs and staff

what revisions to the form and/or the process are needed

Project was funded by IES to meet the urgent need to know more about the valdity of the Child Outcomes Summary Process

Series of studies – focus on the child outcomes summary form

Conditions meaningful and useful COSF data

Impact of use on practice, pos or neg

Revisions or recommendations from that

For other audiences: ECO is a national center located at SRI that has been working with states – first to identify what outcomes to measure and then to support the development of effective systems to measure those outcomes so that data is available to guide decision-making at the state and national levels.

Four ENHANCE Studies

So, let’s unpack this…. What does all this mean?

ENHANCE include 5 studies staggered across 3.5 years,

Studies use different methods to get at questions related to the COSF process.

Comparison COSF ratings with external child assessments

Video tape team decision making process where COSF ratings are decided

Online provider survey – opportunity, if interested, to share your input about the process you’ve observed and its impact on your work

ENHANCE staff come for on-site record reviews to relate COSF info to broader info in the file

Not at this program site, also analyze state COSF data to look for patterns

States if needed: IL, MN, ME, TX, NC, and one other Part C (probably NM)

IL, MN, ME, TX, and two others (NM, TN, VA, KS?)

Studies 1-3: 34 Project Data Collection Sites

17 Part C (Birth to 3)

Illinois

Maine

Minnesota

Illinois

Maine

Minnesota

9

Data is being collected from 36 local programs or school districts in 7 states for the first 4 studies

9

Goals

Program Entry

Program Exit

Sample

Study Status

Recruiting families

About ½ of the sample enrolled

See expected variability in sample (ages, disability types) and initial COS ratings/assessment scores

Comparison with Child Assessments Study

Conclusions about the child’s progress and changes with regard to age-expected functioning

10

Goals

Learn more about the implementation of the COS process, including how the team reaches a decision about a rating and what is discussed.

Do COS ratings assigned match the developmental level of the behaviors presented in the meeting?

What is team understanding of outcomes and rating criteria?

Sample

180 children each from Part C & Part B 619 ½ entry & ½ exit meetings

Study Status

19 videos received

Goals

What is the impact of the COS process on practice?

What have providers learned about the COS?

What else would be helpful?

Sample

participate in the COS process

are invited to participate

12

Goals

Analyze characteristics of COS data and relationships to other variables

Look for consistency in patterns across states

Examples of Questions

Are patterns in COS data across states consistent with those predicted for high quality data?

How are COS ratings related to hypothesized variables (e.g., disability type) and not to other variables (e.g., gender)?

How are team variables related to COS ratings?

Sample

All valid COS data within the state for a reporting year

15-18 states conducting all analyses

Additional states sharing select analyses

State Data Study

Focus of session today is to report on the preliminary findings from the state data study

Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…

Entire state level population on children birth-5 with disabilities served under IDEA

13

Refined procedures for gathering data tables by gathering data from a preliminary group of 6 states

Mostly states used procedures and generated data tables

A few provided formatted data files for SRI to analyze

Beginning to analyze data from that preliminary group

Soon will request data from other states in state data study and permission to use relevant data additional states have already analyzed and shared

State Data Study: Status

Focus of session today is to report on the preliminary findings from the state data study

Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…

14

3 Part C (Birth to 3)

3 Part B Preschool (3-5)

Plug for permission to share data form….

15

de-identified data files OR

aggregate output or reports from a set of requested analyses

Examples of analyses include

relationships between outcomes

relationships between outcomes across time

relationships of outcome scores to other factors such as disability and gender

Similar to what is shown on COSF data CoP

What data would I need to submit?

Data collected at entry and exit from Part C and Part B 619 programs

COSF ratings

Variables that describe the setting or composition of the services

How will I submit data?

De-identified data files

Analyzed data

Submitted through secure server or emailed

Who do I contact for more information?

Cornelia Taylor

Should they differ across outcomes?

Where do most of the ratings fall?

How much should the extremes of the scale be used ( 1 or 7)?

Entry Rating Expectations

Entry Data Analysis

The following data are from 3 Part C programs and 2 Part B programs

All data are from 08 – 09

The data are entry cohorts

i.e. all children who entered during the FFY

Part C 08-09 entry ratings across states; Outcome A

1 2 3 4 5 6 7

24

State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 8.6044318717586069E-2 0.16501650165016507 0.28571428571428581 0.12682696841112684 0.18104667609618108 0.11904761904761907 3.6303630363036313E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.7082452431289656E-2 8.6680761099365733E-2 0.12896405919661735 0.14376321353065541 0.18816067653276958 0.2452431289640592 0.15010570824524314 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.439988541965053E-2 0.1566886279003151 0.20309366943569179 0.14351188771125756 0.21140074477227161 0.1 6642795760527074 7.4477227155542849E-2

Part C 08-09 entry ratings across states; Outcome B

1 2 3 4 5 6 7

State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 9.6888260254596917E-2 0.18552569542668554 0.29820839226779827 0.1478076379066478 0.16030174446016029 8.6515794436586524E-2 2.4752475247524754E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.6511627906976764E-2 0.12262156448202963 0.18181818181818188 0.21775898520084569 0.22410147991543342 0.14376321353065541 6.3424947145877389E-2 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.4139215124606135E-2 0.15812088226869092 0.2142652535090232 6 0.15983958751074193 0.19936980807791466 0.15124606130048698 6.3019192208536245E-2

Part C 08-09 entry ratings across states; Outcome C

1 2 3 4 5 6 7

State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 0.11739745403111741 0.18906176331918906 0.30575200377180578 0.14521452145214525 0.15299387081565299 6.9071192833569064E-2 2.0509193776520517E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.1310782241014807E-2 0.10993657505285413 0.16067653276955596 0.15644820295983092 0.21775898520084569 0.19238900634249476 0.1014799154334038 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.0727585219134919E-2 0.15983958751074193 0.21627040962474936 0.156 6886279003151 0.20739043254081926 0.14694929819535954 5.2134059008879977E-2

Outcome A – Average Entry Ratings

Part C (n =3) 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962188587940747E-2 Part B (n=2) 5.1960960414547751E-2 0.15859403916555473 0.20366799945257805 0.18455838442086803 0.2163961437853609 0.1426524150329542 4.2170057728136569E-2

Outcome B – Average Entry Ratings

Part C (n =3) 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.03988 7153397947E-2 Part B (n=2) 5.3578659282757957E-2 0.14680662837010997 0.20212615389811661 0.19515223141097091 0.21205058704932048 0.13835220555949612 5.1933534429228032E-2

Outcome C – Average Entry Ratings

Part C (n=3) 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.80 41056072934763E-2 Part B (n=2) 4.6688853866971546E-2 0.1119449425262222 0.16157549427240667 0.17067278656449572 0.22167555794704227 0.19617495322022888 9.1267411602632784E-2

The difference in distributions between Part C and Part B are largest for Outcome C

Children in Part B enter with higher ratings

Things to notice

OC-A 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962 188587940747E-2 OC-B 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.039887153397947E-2 OC-C 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.8041056072934763E-2

Things to Notice

More that ½ of all children enter with a COS rating of

3,4 or 5 across outcomes.

An average of 12% of children enter at with the very lowest (1) or the very highest (7) across outcomes.

The typical entry distribution has most children towards the middle of the distribution.

Conclusions Across Part C and Part B

No Action Interpretation: You may be serving a population that is higher or lower functioning that other states.

Pattern Check: if the distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution.

Action Interpretation: Your providers may be systematically misunderstanding the definition of COS rating points.

Correlations between entry ratings

Program

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part C 08-09 exit ratings across states; Outcome B

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part C 08-09 exit ratings across states; Outcome C

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part B 08-09 exit ratings across states; Outcome A

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part B 08-09 exit ratings across states; Outcome B

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part B 08-09 exit ratings across states; Outcome C

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Outcome A – Average Exit Ratings

Part C (n=3) 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.217003516792 05465 Part B (n=3) 1.2463293008708791E-2 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075

Outcome B – Average Exit Ratings

Part C (n=3) 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.22 50263096547733 Part B (n=3) 1.3222461117455763E-2 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277

Outcome C – Average Exit Ratings

Part C (n=3) 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.1 9443906456404905 Part B (n=3) 1.4892746134255596E-2 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879

Part C 08-09 average exit scores across outcomes (state n = 3)

OC-A 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.21700351679205465 OC-B 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.2250263096547733 OC-C 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.19443906456404905

Part B 08-09 average exit scores across outcomes (state n=3)

OC-A 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075 OC-B 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277 OC-C 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879

Variation in ratings across outcomes

The exit distribution is shifted toward a higher rating than is the entry distribution

For Part B, the average percent of children with a rating of 7 is much higher for Outcome C than for the other two outcomes

Things to Notice

No Action Interpretation: You may be serving a lower functioning group than other states

If this interpretation is true, it should also be apparent in your entry distribution

Pattern Check: the distribution of exit scores in your state is not skewed towards the higher end of the rating scale.

Action Interpretation: The children in your programs may not be making expected gains.

Choosing a metric for looking at paired distributions

Progress categories

Side-by-side entry exit comparisons

Both of the above can be completed using the COS calculator 2.0

Entry-Exit Paired Distribution

How many points the child’s rating changed between entry and exit?

What would you expect to see?

Exit rating minus Entry rating

Exit Rating

Entry Rating

-4 -2 0 2 4

Part B exit score – entry score; 08-09

State B Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 3.1028368794326251E-3 1.0195035460992909E-2 1.0195035460992909E-2 2.1719858156028372E-2 0.12101063829787234 0.24202127659574471 0.26684397163120571 0.2021276595744681 9.5744680851063843E-2 2.8812056737588642E-2 7.5354609929078028E-3 State C Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 1.2145748987854249E-2 1.2145748987854249E-2 1.2145748987854249E-2 4.4534412955465591E-2 0.44129554655870434 0.26315789473684215 0.10931174089068826 8.0971659919028327E-2 2.8340080971659919E-2 4.0485829959514179E-3 4.0485829959514179E-3 State D Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 2.019727571629874E-2 3.240958196336309E-2 3.240958196336309E-2 7.2804133395960549E-2 0.29920150305307658 0.26538280883043686 0.16063879755753874 8.0319398778769396E-2 4.8379520901831861E-2 1.1742602160638799E-2 1.8788163457022081E-3

Things to Notice

Most children’s ratings increase 1, 2, or 3 points, or they stay the same

Very few children have ratings that decrease

However, more children have ratings that decrease in Part C than in Part B

No Action Interpretation: Your programs are very effective and children make large gains (verify!).

Pattern Check: if a large percentage of children in your state make large increases in their ratings

Action Interpretation: Providers are not using the scale correctly and may be inflating exit ratings and/or deflating entry ratings.

Additional entry-exit analysis

Progress categories by other variables (e.g., disability type, primary language)

The distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution

The distribution of exit scores in your state is not skewed towards the higher end of the rating scale.

A large percentage of children in your state make large increases in their ratings

Summary of pattern checks

Individual emails for Kathy Hebbeler, Donna Spiker, Lauren Barton, or Cornelia Taylor

58

New Orleans, LA

from the ENHANCE Project

Overview of ENHANCE: Research Underway on the Validity of the Child Outcomes Summary Form (COSF)

Are COSF data valid and reliable? This session will provide an overview of a research project underway to investigate the validity of the COSF. Plans for using information learned to provide better guidance about the COSF and implementing the COSF process will be shared. The group will discuss the kinds of validity information needed by States and specific content that might be important to investigate further in the studies

1

Identify the purpose and approach of the state data study

Describe some preliminary findings from initial states involved in the state data study

Explain how other states could examine their own data in the same way as that presented

Discuss any emerging implications for validity of the COS and for interpreting individual state data

Today’s session

Progress of project efforts to investigate the validity of the Child Outcomes Summary process

2

Origins of ENHANCE

COS Process

Implemented > 40 States, Little Systematic Validation for Use in Accountability

Origins of ENHANCE

COS Process

?

Early Evidence

Belief in potential for COS process to be valid based on:

Existing literature: team-based decision-making can be reliable and valid

Existing literature: teams are effective in identifying individual children’s functioning so that they can plan and deliver appropriate services

Early data from states: pilot sites, small n’s showing similarity in distributions, sensible patterns for subgroups

Anecdotal data from trainers: participants reach decisions fairly easily and consistently

ENHANCE

Project launched by the Early Childhood Outcomes Center (ECO) and SRI International

Funded by the U.S. Dept. of Education, Institute for Educational Sciences – July 1, 2009

Series of studies designed to find out:

the conditions under which the Child Outcomes Summary (COS) Process produces meaningful and useful data for accountability and program improvement

the positive and/or negative impact of the COS process on programs and staff

what revisions to the form and/or the process are needed

Project was funded by IES to meet the urgent need to know more about the valdity of the Child Outcomes Summary Process

Series of studies – focus on the child outcomes summary form

Conditions meaningful and useful COSF data

Impact of use on practice, pos or neg

Revisions or recommendations from that

For other audiences: ECO is a national center located at SRI that has been working with states – first to identify what outcomes to measure and then to support the development of effective systems to measure those outcomes so that data is available to guide decision-making at the state and national levels.

Four ENHANCE Studies

So, let’s unpack this…. What does all this mean?

ENHANCE include 5 studies staggered across 3.5 years,

Studies use different methods to get at questions related to the COSF process.

Comparison COSF ratings with external child assessments

Video tape team decision making process where COSF ratings are decided

Online provider survey – opportunity, if interested, to share your input about the process you’ve observed and its impact on your work

ENHANCE staff come for on-site record reviews to relate COSF info to broader info in the file

Not at this program site, also analyze state COSF data to look for patterns

States if needed: IL, MN, ME, TX, NC, and one other Part C (probably NM)

IL, MN, ME, TX, and two others (NM, TN, VA, KS?)

Studies 1-3: 34 Project Data Collection Sites

17 Part C (Birth to 3)

Illinois

Maine

Minnesota

Illinois

Maine

Minnesota

9

Data is being collected from 36 local programs or school districts in 7 states for the first 4 studies

9

Goals

Program Entry

Program Exit

Sample

Study Status

Recruiting families

About ½ of the sample enrolled

See expected variability in sample (ages, disability types) and initial COS ratings/assessment scores

Comparison with Child Assessments Study

Conclusions about the child’s progress and changes with regard to age-expected functioning

10

Goals

Learn more about the implementation of the COS process, including how the team reaches a decision about a rating and what is discussed.

Do COS ratings assigned match the developmental level of the behaviors presented in the meeting?

What is team understanding of outcomes and rating criteria?

Sample

180 children each from Part C & Part B 619 ½ entry & ½ exit meetings

Study Status

19 videos received

Goals

What is the impact of the COS process on practice?

What have providers learned about the COS?

What else would be helpful?

Sample

participate in the COS process

are invited to participate

12

Goals

Analyze characteristics of COS data and relationships to other variables

Look for consistency in patterns across states

Examples of Questions

Are patterns in COS data across states consistent with those predicted for high quality data?

How are COS ratings related to hypothesized variables (e.g., disability type) and not to other variables (e.g., gender)?

How are team variables related to COS ratings?

Sample

All valid COS data within the state for a reporting year

15-18 states conducting all analyses

Additional states sharing select analyses

State Data Study

Focus of session today is to report on the preliminary findings from the state data study

Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…

Entire state level population on children birth-5 with disabilities served under IDEA

13

Refined procedures for gathering data tables by gathering data from a preliminary group of 6 states

Mostly states used procedures and generated data tables

A few provided formatted data files for SRI to analyze

Beginning to analyze data from that preliminary group

Soon will request data from other states in state data study and permission to use relevant data additional states have already analyzed and shared

State Data Study: Status

Focus of session today is to report on the preliminary findings from the state data study

Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…

14

3 Part C (Birth to 3)

3 Part B Preschool (3-5)

Plug for permission to share data form….

15

de-identified data files OR

aggregate output or reports from a set of requested analyses

Examples of analyses include

relationships between outcomes

relationships between outcomes across time

relationships of outcome scores to other factors such as disability and gender

Similar to what is shown on COSF data CoP

What data would I need to submit?

Data collected at entry and exit from Part C and Part B 619 programs

COSF ratings

Variables that describe the setting or composition of the services

How will I submit data?

De-identified data files

Analyzed data

Submitted through secure server or emailed

Who do I contact for more information?

Cornelia Taylor

Should they differ across outcomes?

Where do most of the ratings fall?

How much should the extremes of the scale be used ( 1 or 7)?

Entry Rating Expectations

Entry Data Analysis

The following data are from 3 Part C programs and 2 Part B programs

All data are from 08 – 09

The data are entry cohorts

i.e. all children who entered during the FFY

Part C 08-09 entry ratings across states; Outcome A

1 2 3 4 5 6 7

24

State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 8.6044318717586069E-2 0.16501650165016507 0.28571428571428581 0.12682696841112684 0.18104667609618108 0.11904761904761907 3.6303630363036313E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.7082452431289656E-2 8.6680761099365733E-2 0.12896405919661735 0.14376321353065541 0.18816067653276958 0.2452431289640592 0.15010570824524314 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.439988541965053E-2 0.1566886279003151 0.20309366943569179 0.14351188771125756 0.21140074477227161 0.1 6642795760527074 7.4477227155542849E-2

Part C 08-09 entry ratings across states; Outcome B

1 2 3 4 5 6 7

State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 9.6888260254596917E-2 0.18552569542668554 0.29820839226779827 0.1478076379066478 0.16030174446016029 8.6515794436586524E-2 2.4752475247524754E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.6511627906976764E-2 0.12262156448202963 0.18181818181818188 0.21775898520084569 0.22410147991543342 0.14376321353065541 6.3424947145877389E-2 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.4139215124606135E-2 0.15812088226869092 0.2142652535090232 6 0.15983958751074193 0.19936980807791466 0.15124606130048698 6.3019192208536245E-2

Part C 08-09 entry ratings across states; Outcome C

1 2 3 4 5 6 7

State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 0.11739745403111741 0.18906176331918906 0.30575200377180578 0.14521452145214525 0.15299387081565299 6.9071192833569064E-2 2.0509193776520517E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.1310782241014807E-2 0.10993657505285413 0.16067653276955596 0.15644820295983092 0.21775898520084569 0.19238900634249476 0.1014799154334038 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.0727585219134919E-2 0.15983958751074193 0.21627040962474936 0.156 6886279003151 0.20739043254081926 0.14694929819535954 5.2134059008879977E-2

Outcome A – Average Entry Ratings

Part C (n =3) 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962188587940747E-2 Part B (n=2) 5.1960960414547751E-2 0.15859403916555473 0.20366799945257805 0.18455838442086803 0.2163961437853609 0.1426524150329542 4.2170057728136569E-2

Outcome B – Average Entry Ratings

Part C (n =3) 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.03988 7153397947E-2 Part B (n=2) 5.3578659282757957E-2 0.14680662837010997 0.20212615389811661 0.19515223141097091 0.21205058704932048 0.13835220555949612 5.1933534429228032E-2

Outcome C – Average Entry Ratings

Part C (n=3) 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.80 41056072934763E-2 Part B (n=2) 4.6688853866971546E-2 0.1119449425262222 0.16157549427240667 0.17067278656449572 0.22167555794704227 0.19617495322022888 9.1267411602632784E-2

The difference in distributions between Part C and Part B are largest for Outcome C

Children in Part B enter with higher ratings

Things to notice

OC-A 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962 188587940747E-2 OC-B 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.039887153397947E-2 OC-C 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.8041056072934763E-2

Things to Notice

More that ½ of all children enter with a COS rating of

3,4 or 5 across outcomes.

An average of 12% of children enter at with the very lowest (1) or the very highest (7) across outcomes.

The typical entry distribution has most children towards the middle of the distribution.

Conclusions Across Part C and Part B

No Action Interpretation: You may be serving a population that is higher or lower functioning that other states.

Pattern Check: if the distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution.

Action Interpretation: Your providers may be systematically misunderstanding the definition of COS rating points.

Correlations between entry ratings

Program

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part C 08-09 exit ratings across states; Outcome B

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part C 08-09 exit ratings across states; Outcome C

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part B 08-09 exit ratings across states; Outcome A

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part B 08-09 exit ratings across states; Outcome B

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Part B 08-09 exit ratings across states; Outcome C

1 2 3 4 5 6 7

State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289

Outcome A – Average Exit Ratings

Part C (n=3) 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.217003516792 05465 Part B (n=3) 1.2463293008708791E-2 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075

Outcome B – Average Exit Ratings

Part C (n=3) 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.22 50263096547733 Part B (n=3) 1.3222461117455763E-2 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277

Outcome C – Average Exit Ratings

Part C (n=3) 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.1 9443906456404905 Part B (n=3) 1.4892746134255596E-2 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879

Part C 08-09 average exit scores across outcomes (state n = 3)

OC-A 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.21700351679205465 OC-B 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.2250263096547733 OC-C 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.19443906456404905

Part B 08-09 average exit scores across outcomes (state n=3)

OC-A 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075 OC-B 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277 OC-C 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879

Variation in ratings across outcomes

The exit distribution is shifted toward a higher rating than is the entry distribution

For Part B, the average percent of children with a rating of 7 is much higher for Outcome C than for the other two outcomes

Things to Notice

No Action Interpretation: You may be serving a lower functioning group than other states

If this interpretation is true, it should also be apparent in your entry distribution

Pattern Check: the distribution of exit scores in your state is not skewed towards the higher end of the rating scale.

Action Interpretation: The children in your programs may not be making expected gains.

Choosing a metric for looking at paired distributions

Progress categories

Side-by-side entry exit comparisons

Both of the above can be completed using the COS calculator 2.0

Entry-Exit Paired Distribution

How many points the child’s rating changed between entry and exit?

What would you expect to see?

Exit rating minus Entry rating

Exit Rating

Entry Rating

-4 -2 0 2 4

Part B exit score – entry score; 08-09

State B Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 3.1028368794326251E-3 1.0195035460992909E-2 1.0195035460992909E-2 2.1719858156028372E-2 0.12101063829787234 0.24202127659574471 0.26684397163120571 0.2021276595744681 9.5744680851063843E-2 2.8812056737588642E-2 7.5354609929078028E-3 State C Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 1.2145748987854249E-2 1.2145748987854249E-2 1.2145748987854249E-2 4.4534412955465591E-2 0.44129554655870434 0.26315789473684215 0.10931174089068826 8.0971659919028327E-2 2.8340080971659919E-2 4.0485829959514179E-3 4.0485829959514179E-3 State D Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 2.019727571629874E-2 3.240958196336309E-2 3.240958196336309E-2 7.2804133395960549E-2 0.29920150305307658 0.26538280883043686 0.16063879755753874 8.0319398778769396E-2 4.8379520901831861E-2 1.1742602160638799E-2 1.8788163457022081E-3

Things to Notice

Most children’s ratings increase 1, 2, or 3 points, or they stay the same

Very few children have ratings that decrease

However, more children have ratings that decrease in Part C than in Part B

No Action Interpretation: Your programs are very effective and children make large gains (verify!).

Pattern Check: if a large percentage of children in your state make large increases in their ratings

Action Interpretation: Providers are not using the scale correctly and may be inflating exit ratings and/or deflating entry ratings.

Additional entry-exit analysis

Progress categories by other variables (e.g., disability type, primary language)

The distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution

The distribution of exit scores in your state is not skewed towards the higher end of the rating scale.

A large percentage of children in your state make large increases in their ratings

Summary of pattern checks

Individual emails for Kathy Hebbeler, Donna Spiker, Lauren Barton, or Cornelia Taylor

58

Related Documents See more >