Patterns in Child Outcomes Summary Data:
Jan 23, 2016
Patterns in Child Outcomes Summary Data:. Analytic Approaches and Early Findings from the ENHANCE Project. Cornelia Taylor, Lauren Barton, Donna Spiker September 19-21, 2011. Measuring and Improving Child and Family Outcomes Conference New Orleans, LA. Today’s session. - PowerPoint PPT Presentation
Slide 1September 19-21, 2011
New Orleans, LA
from the ENHANCE Project
Overview of ENHANCE: Research Underway on the Validity of the Child Outcomes Summary Form (COSF)
Are COSF data valid and reliable? This session will provide an overview of a research project underway to investigate the validity of the COSF. Plans for using information learned to provide better guidance about the COSF and implementing the COSF process will be shared. The group will discuss the kinds of validity information needed by States and specific content that might be important to investigate further in the studies
1
Identify the purpose and approach of the state data study
Describe some preliminary findings from initial states involved in the state data study
Explain how other states could examine their own data in the same way as that presented
Discuss any emerging implications for validity of the COS and for interpreting individual state data
Today’s session
Progress of project efforts to investigate the validity of the Child Outcomes Summary process
2
Origins of ENHANCE
COS Process
Implemented > 40 States, Little Systematic Validation for Use in Accountability
Origins of ENHANCE
COS Process
?
Early Evidence
Belief in potential for COS process to be valid based on:
Existing literature: team-based decision-making can be reliable and valid
Existing literature: teams are effective in identifying individual children’s functioning so that they can plan and deliver appropriate services
Early data from states: pilot sites, small n’s showing similarity in distributions, sensible patterns for subgroups
Anecdotal data from trainers: participants reach decisions fairly easily and consistently
ENHANCE
Project launched by the Early Childhood Outcomes Center (ECO) and SRI International
Funded by the U.S. Dept. of Education, Institute for Educational Sciences – July 1, 2009
Series of studies designed to find out:
the conditions under which the Child Outcomes Summary (COS) Process produces meaningful and useful data for accountability and program improvement
the positive and/or negative impact of the COS process on programs and staff
what revisions to the form and/or the process are needed
Project was funded by IES to meet the urgent need to know more about the valdity of the Child Outcomes Summary Process
Series of studies – focus on the child outcomes summary form
Conditions meaningful and useful COSF data
Impact of use on practice, pos or neg
Revisions or recommendations from that
For other audiences: ECO is a national center located at SRI that has been working with states – first to identify what outcomes to measure and then to support the development of effective systems to measure those outcomes so that data is available to guide decision-making at the state and national levels.
Four ENHANCE Studies
So, let’s unpack this…. What does all this mean?
ENHANCE include 5 studies staggered across 3.5 years,
Studies use different methods to get at questions related to the COSF process.
Comparison COSF ratings with external child assessments
Video tape team decision making process where COSF ratings are decided
Online provider survey – opportunity, if interested, to share your input about the process you’ve observed and its impact on your work
ENHANCE staff come for on-site record reviews to relate COSF info to broader info in the file
Not at this program site, also analyze state COSF data to look for patterns
States if needed: IL, MN, ME, TX, NC, and one other Part C (probably NM)
IL, MN, ME, TX, and two others (NM, TN, VA, KS?)
Studies 1-3: 34 Project Data Collection Sites
17 Part C (Birth to 3)
Illinois
Maine
Minnesota
Illinois
Maine
Minnesota
9
Data is being collected from 36 local programs or school districts in 7 states for the first 4 studies
9
Goals
Program Entry
Program Exit
Sample
Study Status
Recruiting families
About ½ of the sample enrolled
See expected variability in sample (ages, disability types) and initial COS ratings/assessment scores
Comparison with Child Assessments Study
Conclusions about the child’s progress and changes with regard to age-expected functioning
10
Goals
Learn more about the implementation of the COS process, including how the team reaches a decision about a rating and what is discussed.
Do COS ratings assigned match the developmental level of the behaviors presented in the meeting?
What is team understanding of outcomes and rating criteria?
Sample
180 children each from Part C & Part B 619 ½ entry & ½ exit meetings
Study Status
19 videos received
Goals
What is the impact of the COS process on practice?
What have providers learned about the COS?
What else would be helpful?
Sample
participate in the COS process
are invited to participate
12
Goals
Analyze characteristics of COS data and relationships to other variables
Look for consistency in patterns across states
Examples of Questions
Are patterns in COS data across states consistent with those predicted for high quality data?
How are COS ratings related to hypothesized variables (e.g., disability type) and not to other variables (e.g., gender)?
How are team variables related to COS ratings?
Sample
All valid COS data within the state for a reporting year
15-18 states conducting all analyses
Additional states sharing select analyses
State Data Study
Focus of session today is to report on the preliminary findings from the state data study
Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…
Entire state level population on children birth-5 with disabilities served under IDEA
13
Refined procedures for gathering data tables by gathering data from a preliminary group of 6 states
Mostly states used procedures and generated data tables
A few provided formatted data files for SRI to analyze
Beginning to analyze data from that preliminary group
Soon will request data from other states in state data study and permission to use relevant data additional states have already analyzed and shared
State Data Study: Status
Focus of session today is to report on the preliminary findings from the state data study
Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…
14
3 Part C (Birth to 3)
3 Part B Preschool (3-5)
Plug for permission to share data form….
15
de-identified data files OR
aggregate output or reports from a set of requested analyses
Examples of analyses include
relationships between outcomes
relationships between outcomes across time
relationships of outcome scores to other factors such as disability and gender
Similar to what is shown on COSF data CoP
What data would I need to submit?
Data collected at entry and exit from Part C and Part B 619 programs
COSF ratings
Variables that describe the setting or composition of the services
How will I submit data?
De-identified data files
Analyzed data
Submitted through secure server or emailed
Who do I contact for more information?
Cornelia Taylor
Should they differ across outcomes?
Where do most of the ratings fall?
How much should the extremes of the scale be used ( 1 or 7)?
Entry Rating Expectations
Entry Data Analysis
The following data are from 3 Part C programs and 2 Part B programs
All data are from 08 – 09
The data are entry cohorts
i.e. all children who entered during the FFY
Part C 08-09 entry ratings across states; Outcome A
1 2 3 4 5 6 7
24
State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 8.6044318717586069E-2 0.16501650165016507 0.28571428571428581 0.12682696841112684 0.18104667609618108 0.11904761904761907 3.6303630363036313E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.7082452431289656E-2 8.6680761099365733E-2 0.12896405919661735 0.14376321353065541 0.18816067653276958 0.2452431289640592 0.15010570824524314 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.439988541965053E-2 0.1566886279003151 0.20309366943569179 0.14351188771125756 0.21140074477227161 0.1 6642795760527074 7.4477227155542849E-2
Part C 08-09 entry ratings across states; Outcome B
1 2 3 4 5 6 7
State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 9.6888260254596917E-2 0.18552569542668554 0.29820839226779827 0.1478076379066478 0.16030174446016029 8.6515794436586524E-2 2.4752475247524754E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.6511627906976764E-2 0.12262156448202963 0.18181818181818188 0.21775898520084569 0.22410147991543342 0.14376321353065541 6.3424947145877389E-2 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.4139215124606135E-2 0.15812088226869092 0.2142652535090232 6 0.15983958751074193 0.19936980807791466 0.15124606130048698 6.3019192208536245E-2
Part C 08-09 entry ratings across states; Outcome C
1 2 3 4 5 6 7
State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 0.11739745403111741 0.18906176331918906 0.30575200377180578 0.14521452145214525 0.15299387081565299 6.9071192833569064E-2 2.0509193776520517E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.1310782241014807E-2 0.10993657505285413 0.16067653276955596 0.15644820295983092 0.21775898520084569 0.19238900634249476 0.1014799154334038 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.0727585219134919E-2 0.15983958751074193 0.21627040962474936 0.156 6886279003151 0.20739043254081926 0.14694929819535954 5.2134059008879977E-2
Outcome A – Average Entry Ratings
Part C (n =3) 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962188587940747E-2 Part B (n=2) 5.1960960414547751E-2 0.15859403916555473 0.20366799945257805 0.18455838442086803 0.2163961437853609 0.1426524150329542 4.2170057728136569E-2
Outcome B – Average Entry Ratings
Part C (n =3) 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.03988 7153397947E-2 Part B (n=2) 5.3578659282757957E-2 0.14680662837010997 0.20212615389811661 0.19515223141097091 0.21205058704932048 0.13835220555949612 5.1933534429228032E-2
Outcome C – Average Entry Ratings
Part C (n=3) 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.80 41056072934763E-2 Part B (n=2) 4.6688853866971546E-2 0.1119449425262222 0.16157549427240667 0.17067278656449572 0.22167555794704227 0.19617495322022888 9.1267411602632784E-2
The difference in distributions between Part C and Part B are largest for Outcome C
Children in Part B enter with higher ratings
Things to notice
OC-A 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962 188587940747E-2 OC-B 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.039887153397947E-2 OC-C 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.8041056072934763E-2
Things to Notice
More that ½ of all children enter with a COS rating of
3,4 or 5 across outcomes.
An average of 12% of children enter at with the very lowest (1) or the very highest (7) across outcomes.
The typical entry distribution has most children towards the middle of the distribution.
Conclusions Across Part C and Part B
No Action Interpretation: You may be serving a population that is higher or lower functioning that other states.
Pattern Check: if the distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution.
Action Interpretation: Your providers may be systematically misunderstanding the definition of COS rating points.
Correlations between entry ratings
Program
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part C 08-09 exit ratings across states; Outcome B
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part C 08-09 exit ratings across states; Outcome C
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part B 08-09 exit ratings across states; Outcome A
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part B 08-09 exit ratings across states; Outcome B
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part B 08-09 exit ratings across states; Outcome C
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Outcome A – Average Exit Ratings
Part C (n=3) 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.217003516792 05465 Part B (n=3) 1.2463293008708791E-2 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075
Outcome B – Average Exit Ratings
Part C (n=3) 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.22 50263096547733 Part B (n=3) 1.3222461117455763E-2 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277
Outcome C – Average Exit Ratings
Part C (n=3) 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.1 9443906456404905 Part B (n=3) 1.4892746134255596E-2 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879
Part C 08-09 average exit scores across outcomes (state n = 3)
OC-A 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.21700351679205465 OC-B 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.2250263096547733 OC-C 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.19443906456404905
Part B 08-09 average exit scores across outcomes (state n=3)
OC-A 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075 OC-B 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277 OC-C 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879
Variation in ratings across outcomes
The exit distribution is shifted toward a higher rating than is the entry distribution
For Part B, the average percent of children with a rating of 7 is much higher for Outcome C than for the other two outcomes
Things to Notice
No Action Interpretation: You may be serving a lower functioning group than other states
If this interpretation is true, it should also be apparent in your entry distribution
Pattern Check: the distribution of exit scores in your state is not skewed towards the higher end of the rating scale.
Action Interpretation: The children in your programs may not be making expected gains.
Choosing a metric for looking at paired distributions
Progress categories
Side-by-side entry exit comparisons
Both of the above can be completed using the COS calculator 2.0
Entry-Exit Paired Distribution
How many points the child’s rating changed between entry and exit?
What would you expect to see?
Exit rating minus Entry rating
Exit Rating
Entry Rating
-4 -2 0 2 4
Part B exit score – entry score; 08-09
State B Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 3.1028368794326251E-3 1.0195035460992909E-2 1.0195035460992909E-2 2.1719858156028372E-2 0.12101063829787234 0.24202127659574471 0.26684397163120571 0.2021276595744681 9.5744680851063843E-2 2.8812056737588642E-2 7.5354609929078028E-3 State C Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 1.2145748987854249E-2 1.2145748987854249E-2 1.2145748987854249E-2 4.4534412955465591E-2 0.44129554655870434 0.26315789473684215 0.10931174089068826 8.0971659919028327E-2 2.8340080971659919E-2 4.0485829959514179E-3 4.0485829959514179E-3 State D Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 2.019727571629874E-2 3.240958196336309E-2 3.240958196336309E-2 7.2804133395960549E-2 0.29920150305307658 0.26538280883043686 0.16063879755753874 8.0319398778769396E-2 4.8379520901831861E-2 1.1742602160638799E-2 1.8788163457022081E-3
Things to Notice
Most children’s ratings increase 1, 2, or 3 points, or they stay the same
Very few children have ratings that decrease
However, more children have ratings that decrease in Part C than in Part B
No Action Interpretation: Your programs are very effective and children make large gains (verify!).
Pattern Check: if a large percentage of children in your state make large increases in their ratings
Action Interpretation: Providers are not using the scale correctly and may be inflating exit ratings and/or deflating entry ratings.
Additional entry-exit analysis
Progress categories by other variables (e.g., disability type, primary language)
The distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution
The distribution of exit scores in your state is not skewed towards the higher end of the rating scale.
A large percentage of children in your state make large increases in their ratings
Summary of pattern checks
Individual emails for Kathy Hebbeler, Donna Spiker, Lauren Barton, or Cornelia Taylor
58
New Orleans, LA
from the ENHANCE Project
Overview of ENHANCE: Research Underway on the Validity of the Child Outcomes Summary Form (COSF)
Are COSF data valid and reliable? This session will provide an overview of a research project underway to investigate the validity of the COSF. Plans for using information learned to provide better guidance about the COSF and implementing the COSF process will be shared. The group will discuss the kinds of validity information needed by States and specific content that might be important to investigate further in the studies
1
Identify the purpose and approach of the state data study
Describe some preliminary findings from initial states involved in the state data study
Explain how other states could examine their own data in the same way as that presented
Discuss any emerging implications for validity of the COS and for interpreting individual state data
Today’s session
Progress of project efforts to investigate the validity of the Child Outcomes Summary process
2
Origins of ENHANCE
COS Process
Implemented > 40 States, Little Systematic Validation for Use in Accountability
Origins of ENHANCE
COS Process
?
Early Evidence
Belief in potential for COS process to be valid based on:
Existing literature: team-based decision-making can be reliable and valid
Existing literature: teams are effective in identifying individual children’s functioning so that they can plan and deliver appropriate services
Early data from states: pilot sites, small n’s showing similarity in distributions, sensible patterns for subgroups
Anecdotal data from trainers: participants reach decisions fairly easily and consistently
ENHANCE
Project launched by the Early Childhood Outcomes Center (ECO) and SRI International
Funded by the U.S. Dept. of Education, Institute for Educational Sciences – July 1, 2009
Series of studies designed to find out:
the conditions under which the Child Outcomes Summary (COS) Process produces meaningful and useful data for accountability and program improvement
the positive and/or negative impact of the COS process on programs and staff
what revisions to the form and/or the process are needed
Project was funded by IES to meet the urgent need to know more about the valdity of the Child Outcomes Summary Process
Series of studies – focus on the child outcomes summary form
Conditions meaningful and useful COSF data
Impact of use on practice, pos or neg
Revisions or recommendations from that
For other audiences: ECO is a national center located at SRI that has been working with states – first to identify what outcomes to measure and then to support the development of effective systems to measure those outcomes so that data is available to guide decision-making at the state and national levels.
Four ENHANCE Studies
So, let’s unpack this…. What does all this mean?
ENHANCE include 5 studies staggered across 3.5 years,
Studies use different methods to get at questions related to the COSF process.
Comparison COSF ratings with external child assessments
Video tape team decision making process where COSF ratings are decided
Online provider survey – opportunity, if interested, to share your input about the process you’ve observed and its impact on your work
ENHANCE staff come for on-site record reviews to relate COSF info to broader info in the file
Not at this program site, also analyze state COSF data to look for patterns
States if needed: IL, MN, ME, TX, NC, and one other Part C (probably NM)
IL, MN, ME, TX, and two others (NM, TN, VA, KS?)
Studies 1-3: 34 Project Data Collection Sites
17 Part C (Birth to 3)
Illinois
Maine
Minnesota
Illinois
Maine
Minnesota
9
Data is being collected from 36 local programs or school districts in 7 states for the first 4 studies
9
Goals
Program Entry
Program Exit
Sample
Study Status
Recruiting families
About ½ of the sample enrolled
See expected variability in sample (ages, disability types) and initial COS ratings/assessment scores
Comparison with Child Assessments Study
Conclusions about the child’s progress and changes with regard to age-expected functioning
10
Goals
Learn more about the implementation of the COS process, including how the team reaches a decision about a rating and what is discussed.
Do COS ratings assigned match the developmental level of the behaviors presented in the meeting?
What is team understanding of outcomes and rating criteria?
Sample
180 children each from Part C & Part B 619 ½ entry & ½ exit meetings
Study Status
19 videos received
Goals
What is the impact of the COS process on practice?
What have providers learned about the COS?
What else would be helpful?
Sample
participate in the COS process
are invited to participate
12
Goals
Analyze characteristics of COS data and relationships to other variables
Look for consistency in patterns across states
Examples of Questions
Are patterns in COS data across states consistent with those predicted for high quality data?
How are COS ratings related to hypothesized variables (e.g., disability type) and not to other variables (e.g., gender)?
How are team variables related to COS ratings?
Sample
All valid COS data within the state for a reporting year
15-18 states conducting all analyses
Additional states sharing select analyses
State Data Study
Focus of session today is to report on the preliminary findings from the state data study
Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…
Entire state level population on children birth-5 with disabilities served under IDEA
13
Refined procedures for gathering data tables by gathering data from a preliminary group of 6 states
Mostly states used procedures and generated data tables
A few provided formatted data files for SRI to analyze
Beginning to analyze data from that preliminary group
Soon will request data from other states in state data study and permission to use relevant data additional states have already analyzed and shared
State Data Study: Status
Focus of session today is to report on the preliminary findings from the state data study
Patterns – in the distributions (see variability, expected amount of progress, etc), in the relationships with other variable, with state factors like % served, patterns in relationships of data between outcomes…
14
3 Part C (Birth to 3)
3 Part B Preschool (3-5)
Plug for permission to share data form….
15
de-identified data files OR
aggregate output or reports from a set of requested analyses
Examples of analyses include
relationships between outcomes
relationships between outcomes across time
relationships of outcome scores to other factors such as disability and gender
Similar to what is shown on COSF data CoP
What data would I need to submit?
Data collected at entry and exit from Part C and Part B 619 programs
COSF ratings
Variables that describe the setting or composition of the services
How will I submit data?
De-identified data files
Analyzed data
Submitted through secure server or emailed
Who do I contact for more information?
Cornelia Taylor
Should they differ across outcomes?
Where do most of the ratings fall?
How much should the extremes of the scale be used ( 1 or 7)?
Entry Rating Expectations
Entry Data Analysis
The following data are from 3 Part C programs and 2 Part B programs
All data are from 08 – 09
The data are entry cohorts
i.e. all children who entered during the FFY
Part C 08-09 entry ratings across states; Outcome A
1 2 3 4 5 6 7
24
State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 8.6044318717586069E-2 0.16501650165016507 0.28571428571428581 0.12682696841112684 0.18104667609618108 0.11904761904761907 3.6303630363036313E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.7082452431289656E-2 8.6680761099365733E-2 0.12896405919661735 0.14376321353065541 0.18816067653276958 0.2452431289640592 0.15010570824524314 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.439988541965053E-2 0.1566886279003151 0.20309366943569179 0.14351188771125756 0.21140074477227161 0.1 6642795760527074 7.4477227155542849E-2
Part C 08-09 entry ratings across states; Outcome B
1 2 3 4 5 6 7
State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 9.6888260254596917E-2 0.18552569542668554 0.29820839226779827 0.1478076379066478 0.16030174446016029 8.6515794436586524E-2 2.4752475247524754E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 4.6511627906976764E-2 0.12262156448202963 0.18181818181818188 0.21775898520084569 0.22410147991543342 0.14376321353065541 6.3424947145877389E-2 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 5.4139215124606135E-2 0.15812088226869092 0.2142652535090232 6 0.15983958751074193 0.19936980807791466 0.15124606130048698 6.3019192208536245E-2
Part C 08-09 entry ratings across states; Outcome C
1 2 3 4 5 6 7
State A Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 0.11739745403111741 0.18906176331918906 0.30575200377180578 0.14521452145214525 0.15299387081565299 6.9071192833569064E-2 2.0509193776520517E-2 State B Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.1310782241014807E-2 0.10993657505285413 0.16067653276955596 0.15644820295983092 0.21775898520084569 0.19238900634249476 0.1014799154334038 State D Average of %COSF -1 Average of %COSF -2 Average of %COSF -3 Average of %COSF -4 Average of %COSF -5 Average of %COSF -6 Average of %COSF -7 6.0727585219134919E-2 0.15983958751074193 0.21627040962474936 0.156 6886279003151 0.20739043254081926 0.14694929819535954 5.2134059008879977E-2
Outcome A – Average Entry Ratings
Part C (n =3) 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962188587940747E-2 Part B (n=2) 5.1960960414547751E-2 0.15859403916555473 0.20366799945257805 0.18455838442086803 0.2163961437853609 0.1426524150329542 4.2170057728136569E-2
Outcome B – Average Entry Ratings
Part C (n =3) 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.03988 7153397947E-2 Part B (n=2) 5.3578659282757957E-2 0.14680662837010997 0.20212615389811661 0.19515223141097091 0.21205058704932048 0.13835220555949612 5.1933534429228032E-2
Outcome C – Average Entry Ratings
Part C (n=3) 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.80 41056072934763E-2 Part B (n=2) 4.6688853866971546E-2 0.1119449425262222 0.16157549427240667 0.17067278656449572 0.22167555794704227 0.19617495322022888 9.1267411602632784E-2
The difference in distributions between Part C and Part B are largest for Outcome C
Children in Part B enter with higher ratings
Things to notice
OC-A 6.2508885522842075E-2 0.13612863021661525 0.20592400478219833 0.13803402321767988 0.19353603246707413 0.17690623520564966 8.6962 188587940747E-2 OC-B 6.5846367762059904E-2 0.15542271405913538 0.23143060919833441 0.17513540353941182 0.19459101081783614 0.12717502308924294 5.039887153397947E-2 OC-C 7.9811940497089062E-2 0.15294597529426174 0.2275663153887037 0.15278378410409713 0.192714429519106 0.13613649912380774 5.8041056072934763E-2
Things to Notice
More that ½ of all children enter with a COS rating of
3,4 or 5 across outcomes.
An average of 12% of children enter at with the very lowest (1) or the very highest (7) across outcomes.
The typical entry distribution has most children towards the middle of the distribution.
Conclusions Across Part C and Part B
No Action Interpretation: You may be serving a population that is higher or lower functioning that other states.
Pattern Check: if the distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution.
Action Interpretation: Your providers may be systematically misunderstanding the definition of COS rating points.
Correlations between entry ratings
Program
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part C 08-09 exit ratings across states; Outcome B
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part C 08-09 exit ratings across states; Outcome C
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part B 08-09 exit ratings across states; Outcome A
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part B 08-09 exit ratings across states; Outcome B
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Part B 08-09 exit ratings across states; Outcome C
1 2 3 4 5 6 7
State A Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 2.8252095622477491E-2 3.6945048121701352E-2 6.1161130083824893E-2 9.469108972368831E-2 0.18689847873331267 0.32101831729276636 0.27103384042222906 State D Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 5.7385759829968125E-3 1.7215727948990435E-2 3.974495217853348E-2 7.1625929861849097E-2 0.16663124335812968 0.32008501594048894 0.3789585547290118 State E Average of %1 Average of %2 Average of %3 Average of %4 Average of %5 Average of %6 Average of %7 1.0687566797292485E-2 2.20876380477378E-2 4.6312789454934104E-2 6.412540078375488E-2 0.15496971856074107 0.3124332027075169 0.38938368364802289
Outcome A – Average Exit Ratings
Part C (n=3) 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.217003516792 05465 Part B (n=3) 1.2463293008708791E-2 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075
Outcome B – Average Exit Ratings
Part C (n=3) 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.22 50263096547733 Part B (n=3) 1.3222461117455763E-2 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277
Outcome C – Average Exit Ratings
Part C (n=3) 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.1 9443906456404905 Part B (n=3) 1.4892746134255596E-2 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879
Part C 08-09 average exit scores across outcomes (state n = 3)
OC-A 1.7162435121344289E-2 4.8186377632981815E-2 9.2462927850789009E-2 0.11652437590272328 0.22589122012589238 0.28276914657421465 0.21700351679205465 OC-B 1.7667394774540556E-2 4.8142308174518415E-2 8.4762084937380036E-2 0.12294831493224577 0.22089193911737223 0.28056164840916975 0.2250263096547733 OC-C 2.2741372150981695E-2 4.333113014119172E-2 9.0208359573163588E-2 0.10805252858214169 0.22598098704601516 0.31524655794245732 0.19443906456404905
Part B 08-09 average exit scores across outcomes (state n=3)
OC-A 3.093625882722096E-2 7.4867338411775397E-2 0.11711358392623959 0.24697736595415631 0.31502924748859146 0.2026129123833075 OC-B 3.6311076217773602E-2 7.3823749957329932E-2 0.115778277232953 0.24294122326873605 0.32581164011672897 0.19211157208902277 OC-C 2.5416138039476531E-2 4.9072957239097499E-2 7.6814140123097438E-2 0.16949981355072782 0.31784551198025746 0.3464586929330879
Variation in ratings across outcomes
The exit distribution is shifted toward a higher rating than is the entry distribution
For Part B, the average percent of children with a rating of 7 is much higher for Outcome C than for the other two outcomes
Things to Notice
No Action Interpretation: You may be serving a lower functioning group than other states
If this interpretation is true, it should also be apparent in your entry distribution
Pattern Check: the distribution of exit scores in your state is not skewed towards the higher end of the rating scale.
Action Interpretation: The children in your programs may not be making expected gains.
Choosing a metric for looking at paired distributions
Progress categories
Side-by-side entry exit comparisons
Both of the above can be completed using the COS calculator 2.0
Entry-Exit Paired Distribution
How many points the child’s rating changed between entry and exit?
What would you expect to see?
Exit rating minus Entry rating
Exit Rating
Entry Rating
-4 -2 0 2 4
Part B exit score – entry score; 08-09
State B Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 3.1028368794326251E-3 1.0195035460992909E-2 1.0195035460992909E-2 2.1719858156028372E-2 0.12101063829787234 0.24202127659574471 0.26684397163120571 0.2021276595744681 9.5744680851063843E-2 2.8812056737588642E-2 7.5354609929078028E-3 State C Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 1.2145748987854249E-2 1.2145748987854249E-2 1.2145748987854249E-2 4.4534412955465591E-2 0.44129554655870434 0.26315789473684215 0.10931174089068826 8.0971659919028327E-2 2.8340080971659919E-2 4.0485829959514179E-3 4.0485829959514179E-3 State D Average of %_diff_-6 Average of %_diff_-5 Average of %_diff_-4 Average of %_diff_-3 Average of %_diff_-2 Average of %_diff_-1 Average of %_diff_0 Average of %_diff_1 Average of %_diff_2 Average of %_diff_3 Average of %_diff_4 Average of %_diff_5 Average of %_diff_6 0 0 2.019727571629874E-2 3.240958196336309E-2 3.240958196336309E-2 7.2804133395960549E-2 0.29920150305307658 0.26538280883043686 0.16063879755753874 8.0319398778769396E-2 4.8379520901831861E-2 1.1742602160638799E-2 1.8788163457022081E-3
Things to Notice
Most children’s ratings increase 1, 2, or 3 points, or they stay the same
Very few children have ratings that decrease
However, more children have ratings that decrease in Part C than in Part B
No Action Interpretation: Your programs are very effective and children make large gains (verify!).
Pattern Check: if a large percentage of children in your state make large increases in their ratings
Action Interpretation: Providers are not using the scale correctly and may be inflating exit ratings and/or deflating entry ratings.
Additional entry-exit analysis
Progress categories by other variables (e.g., disability type, primary language)
The distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution
The distribution of exit scores in your state is not skewed towards the higher end of the rating scale.
A large percentage of children in your state make large increases in their ratings
Summary of pattern checks
Individual emails for Kathy Hebbeler, Donna Spiker, Lauren Barton, or Cornelia Taylor
58
Related Documents See more >
