Validity of the Child Outcomes Summary Process: Lauren Barton, Cornelia Taylor, Donna Spiker, Kathy Hebbler September 15-17, 2013 Improving Data, Improving Outcomes Conference Washington, DC Updates from the ENHANCE Project
Feb 22, 2016
Validity of the Child Outcomes Summary
Process:
Lauren Barton, Cornelia Taylor, Donna Spiker, Kathy Hebbler
September 15-17, 2013
Improving Data, Improving Outcomes ConferenceWashington, DC
Updates from the ENHANCE Project
• Brief overview of ENHANCE project• Update on status of each study and project-related
resources• Describe some preliminary findings from the child
assessments study• Discuss implications and potential state activities
surrounding reliability and validity of COS process
Today’s session
States identified need…
Origin of ENHANCE
ENHANCE
Series of studies designed to find out:– the conditions under which the Child Outcomes Summary
(COS) Process produces meaningful and useful data for accountability and program improvement
– the positive and/or negative impact of the COS process on programs and staff
– what revisions to the form and/or the process are needed
Four ENHANCE Studies
1) Comparison with Child Assessments
2) Team Decision-Making3) Provider Survey4) State Data Study
Studies 1-3:Project Data Collection Sites
Part C (Birth to 3)• Illinois• Maine• Minnesota• New Mexico• Texas • North Carolina• Virginia
Part B Preschool (3-5)• Illinois• Maine• Minnesota• New Mexico• Texas • South Carolina
6
Get the Survey• http://enhance.sri.com/datacollection/data.html or• ECO website – last year’s conference handouts (ECO resources,
presentations, 2012)• Related presentations posted on same web sitesContent• Training and knowledge of providers• How COS process is structured • Frequency of implementing recommended practices• Attitudes and implementation challenges• Perceived accuracy of the ratings• Impact of COS on practice
Provider Survey
Sample/Approach• All providers in the program who
participate in the COS process are invited to participate in an online survey
Study Status• Data collected spring 2012• Initial analyses completed• 856 providers, 19 EI and 15 ECSE programs in 8 states
Provider Survey
Selected Provider Survey Findings
• Most receive COS training, variable length
• Providers report understanding key concepts
• ¾ of providers complete most ratings in teams
• Few think COS process has a negative impact on providers’ work
• Providers receive limited monitoring, feedback, and support
• Many providers don’t understand why data are collected and what happens with it
• For 1/3 of providers, family input is not included in most COS ratings
Team Decision-Making Study
Approach• Videotape team meetings where COS completed• Code videos for quality indicatorsStudy Status• 131 videos received from 13 EI and 9 ECSE sites in 7 states • Video coding is underwayLearning• About the implementation of the COS process,
including how the team reaches a decision about a rating and what is discussed.
• About team understanding of outcomes, rating criteria, and emphasis on describing child’s functioning
• Gathering information for future guidance
• http://enhance.sri.com/datacollection/data.html 1 page overview of content being coded
in Team Decision-Making videos Paper version of coding form in use
• Codebook with specific information for coding – contact [email protected]
Useful Resources and Additional Information
Approach• Analyze characteristics of COS data
and relationships to other variables• Look for consistency in patterns across states• Examining data relative to first few claims shows some
differences between EI and ECSE data. • Support found in both programs for relative stability in
summary statements year to year.
State Data Study
Sample and Status• All valid COS data within the state for
a reporting year • 9 Part B preschool and 6 Part C states have submitted
their data or a series of analyses for comparison• Additional states sharing select analyses as they do them
anyway for other purposes• Still accepting data from states. Participation involves
sharing the data set used for APR analyses or a series of tables.
State Data Study
• Instruments Abilities Index Child Outcomes Summary form BDI-2 Vineland-II
• Approach Compare COS ratings to BDI-2, Vineland-II scores
oProgram EntryoProgram Exit
Compare conclusions from COS and assessments
Comparison with Child Assessments
Looking Closely at the Study Design…
Might Expect to Compare COS to a “Gold Standard”
Key Differences from Other Concurrent Validity Study Mean…
No gold standard in this case…
Reasons COS and BDI-2 Would or Would Not Show Agreement
Would Agree• Overlapping content being
assessed• Same child, similar time frame• Same family provides input in
both approaches
Would Not Agree• Tools do not reflect the same
content• BDI-2 uses domains; COS uses
outcomes organization• Threshold variation for level of
same-age peers and movement toward that level
• Differences in emphasis on multiple settings and situations
• Single score vs. multiple sources of information
How well do you think progress categories would map on to each other using different approaches (BDI-2 and COS)? Similar/different for…
Children with various types of disabilities/patterns (e.g., younger children)
Types of progress/ratings where expect more/less agreement (e.g., children close to typical developmental levels)
Others?
Expected Agreement Across Tools
• Current Sample 154 children (95 EI, 54 ECSE) Entry data 51 children (31 EI, 16 ECSE) Entry-Exit data
• Study Status Continue data collection through December See expected variability in sample (ages, disability types)
and initial COS ratings/assessment scores
• Today’s Focus Preliminary findings – longitudinal COS-BDI-2 data 51 children (31 EI, 16 ECSE) Entry-Exit data
Comparison with Child Assessments
Sample Characteristics
69%
31%
Children with BDI-2 and COS data at both entry
and exit (n=51)
EI (n = 35)
ECSE (n = 16)
10%
18%
41%
16%
16%
Age at entry (n=51)
Less than 8 months (n = 5)
8-20 months (n = 9)
21-35 months (n= 21)
36-47 months (n = 8)
48 months or more (n = 8)
Mean AgeOverall = 29 months (SD = 16.3, Range 3-63)EI = 20 months (SD = 9.1, Range 3-34)ECSE = 49.4 months (SD = 8..3, Range 36-63)
Sample Characteristics
EI ECSE0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.229
77%
38%44%
19%
Disability Type
Diagnosed condition (n = 8) Developmental delay (EI n = 27; ECSE n = 6)Speech/language impairment (n = 7) Other (n = 3)
Perc
enta
ge o
f Chi
ldre
n
Sample Characteristics
EI
ECSE
Total
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
46%
69%
53%
54%
31%
47%
Gender (n=51)
MaleFemale
Functioning at Entry - ABILITIES IndexSocial/Communication
BEHAVIOR & SOCIAL SKILLS• Social skills• Inappropriate behavior
INTENTIONAL COMMUNICATION• Understanding Others• Communicating with Others
Cognitive/Communication
INTELLECTUAL FUNCTIONING• Thinking and reasoning
INTENTIONAL COMMUNICATION• Understanding others• Communicating with others
Structural Integrity
LIMBS• Use of each hand, arm, & legSTRUCTURAL STATUS• Shape, body form, and
structure
TONICITY• Degree of tightness• Degree of looseness
Integrity of Physical Health
OVERALL HEALTH
Sensory Capabilities
AUDITION• Left and right ears
VISION• Left and right eyes
Severity of Disability at Entry Overall Sample (n = 51)
Overall Sample (n = 51)Possible
MinPossible
Max Mean SD MedianActual
MinimumActual
Maximum
Social/Communication 4 24 9.3 4.2 9 4 21
Cognitive/Communication 3 18 7.2 3.3 7 3 16
Structural Integrity 8 48 12.8 5.8 11 9 36
Integrity of Physical Health 1 6 1.4 1.0 1 1 5
Sensory Capabilities 4 24 4.8 1.9 4 4 12
Total 19 114 30.0 8.5 28 20 59
COS Rating Distributions – Entry (n=51)
Positi
ve S
ocia
l Rel
ation
ship
s
Know
ledg
e an
d Sk
ills
Actio
ns to
Mee
t Nee
dsEntry - Overall (percentages n=51)
0
5
10
15
20
25
30
35
40
45
50
2 4 26 8
48
128
18
24 2422
31
3939
22
14
6
0
10 1234567
Perc
enta
ge o
f Chi
ldre
n
COS Distributions – Exit (n=51)
Positi
ve S
ocia
l Rel
ation
ship
s
Know
ledg
e an
d Sk
ills
Actio
ns to
Mee
t Nee
ds
Exit - Overall (percentages n=51)
0
5
10
15
20
25
30
35
40
45
50
2 2 22 2 24 26
1014
6
2024 22
2731
25
35
25
37
1234567Pe
rcen
tage
of C
hild
ren
COS Distributions – Entry ExitOverall
Outcome 1: Positive Social Relationships
Crosstab showing numbers of children by rating at entry and exit
Exit
Entry 1 2 3 4 5 6 7Grand Total
1 1 0 0 0 0 0 0 1
2 0 0 0 1 0 1 1 3
3 0 0 2 1 0 0 1 4
4 0 1 0 1 5 1 1 9
5 0 0 0 1 3 2 5 11
6 0 0 0 1 2 10 7 20
7 0 0 0 0 0 0 3 3
Grand Total 1 1 2 5 10 14 18 51
COS Distributions – Entry Exit OverallOutcome 2:Crosstab showing numbers of children by rating at entry and exit
Exit
Entry 1 2 3 4 5 6 7Grand Total
1 1 0 0 1 0 0 0 22 0 0 0 2 0 1 1 43 0 1 0 1 2 1 1 64 0 0 0 3 4 3 2 125 0 0 1 0 5 6 4 166 0 0 0 0 1 5 5 117 0 0 0 0 0 0 0 0
Grand Total 1 1 1 7 12 16 13 51
COS Distributions – Entry Exit OverallOutcome 3:Crosstab showing numbers of children by rating at entry and exit
Exit
Entry 1 2 3 4 5 6 7Grand Total
1 1 0 0 0 0 0 0 1
2 0 0 1 0 0 0 1 2
3 0 0 1 1 1 0 1 4
4 0 0 1 2 4 3 2 12
5 0 1 0 0 6 6 7 20
6 0 0 0 0 0 3 4 7
7 0 0 0 0 0 1 4 5
Grand Total 1 1 3 3 11 13 19 51
Methods
• BDI-2 subdomain mapping Positive social relationships: Personal Social Knowledge and skills: Communication & Cognitive Action to meet needs: Adaptive & Motor
Methods – Identifying Progress Categories on the BDI-2
Entry Standard Score
One or Both Subdomains
Exit Standard Score
One Both Subdomains
Change Standard Score
One or Both Subdomains
Raw Score Increase
e > 84 Both > 84 Both
d < 84 One > 84 Both > 5 pts One
c < 84 One < 84 Both > 5 pts One
b < 84 One < 84 Both < 5 pts Both Yes
a < 84 One < 84 Both < 5 pts Both No
Positive Social Relationships (n = 51)
a b c d e0
10
20
30
40
50
60
0.0
22.0
14.0
24.0
40.0
2.0
27.5
7.811.8
51.0
COS %BDI-2 %
Progress Category
Perc
enta
ge o
f Chi
ldre
n
= 5.5
Instrument SS1 SS2BDI-2
40% 63%COS
63% 64%Difference 23% (p>.05) 1.3%(p>.05)
Knowledge and Skills(n = 51)
a b c d e0
10
20
30
40
50
60
0.0
22.020.0
38.0
20.0
3.9
29.433.3
11.8
21.6 COS %BDI-2 %
Progress Category
Perc
enta
ge o
f Chi
ldre
n
Instrument SS1 SS2
BDI-2 57.5% 33.3%
COS 72.5% 58.0%
Difference 15% (p>.05) 25% (p<.05)
= 12.2*
Taking Action to Meet Needs (n = 51)
a b c d e0
10
20
30
40
50
60
0.0
18.8
14.6
41.7
25.0
0.0
33.3
21.6
15.7
29.4
COS %BDI-2 %
Progress Category
Perc
enta
ge o
f Chi
ldre
n
Instrument SS1 SS2
BDI-2 52.8% 45.1%
COS 75.0% 66.7%
Difference 22% (p<.05) 22% (p<.05)
= 8.74*
Child-Level Comparisons of Progress Categories
Child-level comparison of progress categories
a - BDI2 b - BDI2 c - BDI2 d - BDI2 e - BDI2COS - totals
a - COS 0 0 0 0 0 0
b - COS 1 4 3 0 3 11
c - COS 1 3 4 0 2 10
d - COS 0 4 7 4 4 19
e - COS 0 3 3 2 2 10
I - COS 1 1
BDI-Totals 2 15 17 6 11 51
Knowledge and Skills – “Charlie B.”
Charlie B.
• Entry assessment completed at 22 months
• Exit assessment completed 11 months later when Charlie was 33 months
• Diagnosed with a developmental delay• On the COS Knowledge and Skills
– entry rating 5 – exit rating also a 5
• Progress category on COS “b”• Progress category on BDI-2 “e”
“Charlie B.” Qualitative description Exit, 33 months
• Family is concerned because he rarely strings three words together and does not initiate using words.
• Charlie B will answer yes and no questions. • Names colors and refers to himself by a pronoun. • He is beginning to follow two-step unrelated commands
and identify more body parts.• He will attend to adult directed activities. He will attend
to activities when spoken to by an adult. • He identifies matches and sorts colors and shapes. He will
respond to size concepts of big and little. • He maintains focus on activities without becoming overly
distracted.
Standard scores and COS ratings – “Charlie B.”
Entry Exit
Communication 84 88
Cognitive 107 95
COS rating 5 5
EntryExitOnly administered at exit
Mean
- 1 SD
+1 SD
+2 SD
- 2 SD
Reactions?
Child-level comparison of progress categories Knowledge and Skills – Michael J.
a - BDI2 b - BDI2 c - BDI2 d - BDI2 e - BDI2COS - totals
a - COS 0 0 0 0 0 0
b - COS 1 4 3 0 3 11
c - COS 1 3 4 0 2 10
d - COS 0 4 7 4 4 19
e - COS 0 3 3 2 2 10
I - COS 1 1
BDI-Totals 2 15 17 6 11 51
“Michael J.”
• Entry assessment completed at 29 months
• Exit assessment completed 8 months later when Michael was 37 months
• Diagnosed with a developmental delay• On the COS Knowledge and Skills
– entry rating 4 – exit rating also a 5
• Progress category on COS “c”• Progress category on BDI-2 “e”
Michael J. Qualitative descriptionEntry, 29 months
• He responds to a variety of directions and knows his name.
• He follows two-part directions. • He uses gestures and vocalizations to get his
wants and needs met. • His speech is difficult to understand. His
mother reports that others only understand him 10% of the time.
“Michael J.” Qualitative description Exit, 37 months
• He is learning numbers, letter and colors. • He attempts to use words and phrases to
communicate with others. • He has recently had a vocabulary explosion. He
uses 1 – 7 word sentences. • His articulation errors make it difficult for others to
understand him. • His mom estimates that she understands 80-85%
of his speech when the context is readily available.
Standard scores and COS ratings – Michael J.
Entry Exit
Communication 100 102
Cognitive 87 109
COS rating 4 5
Entry
Exit
Reactions?
Child-level comparison of progress categories Knowledge and Skills – Jenny A.
a - BDI2 b - BDI2 c - BDI2 d - BDI2 e - BDI2COS - totals
a - COS 0 0 0 0 0 0
b - COS 1 4 3 0 3 11
c - COS 1 3 4 0 2 10
d - COS 0 4 7 4 4 19
e - COS 0 3 3 2 2 10
I - COS 1 1
BDI-Totals 2 15 17 6 11 51
“Jenny A.”
• Entry assessment completed at 34 months
• Exit assessment completed 10 months later when Jenny was 43 months
• Diagnosed with a developmental delay• On the COS Knowledge and Skills
– entry rating 6 – exit rating a 7
• Progress category on COS “e”• Progress category on BDI-2 “c”
“Jenny A.” Qualitative descriptionEntry, 34 months
• She is moderately intelligible.• She attends to activities for a long time. • She is persistent in figuring things out and has
good creative play.• Parents are not worried about her cognitive
abilities but they are worried that she is harder to understand than her brothers were.
“Jenny A.” Qualitative descriptionExit, 43 months
• There are still concerns about her speech intelligibility.
• Her mother said that she seems to be on task in her skills and that she is understanding more of her communication.
• She is using language more with less frustration. • She is very interested in books and following a story. • She seeks out movement and deep pressure to help
with self-regulation.
Standard scores and COS ratings – Jenny A.
Entry Exit
Communication 85 102
Cognitive 77 82
COS rating 6 7
Entry
Exit
Reactions?
• Preliminary data suggests some similarity, but many children would be classified differently using the 2 methods
• Clinically complex. Different categories but a lot of similarities in the clinical picture.
• Disagreement about what % and actions reflect typical functioning, breadth of sample observations to know functioning…
• Continue to collect more data, investigate further.
Summary
Questions? Comments? Reactions?
Find out more
• ENHANCE Website– http://ENHANCE.sri.com
• ECO Center Website– http://www.the-ECO-center.org
• Contact ENHANCE staff– E-mail: [email protected]