The Current Status of States' Early Childhood Outcome Measurement Systems Kathy Hebbeler, SRI International Lynne Kahn, FPG Child Dev Inst October 17, 2009 DEC's 25th Annual International 1
Dec 31, 2015
The Current Status of States' Early Childhood Outcome Measurement Systems
Kathy Hebbeler, SRI International
Lynne Kahn, FPG Child Dev Inst
October 17, 2009
DEC's 25th Annual International Conference on Young Children with Special Needs and Their Families 1
We’ve come a long way, BABY!
2
In the Beginning- Federal Accountability
• 1994 Government Performance and Results Act (GPRA)
• 2002 Performance Assessment Rating Tool (PART)– Both Part C and Part B Preschool categorized as
“Results Not Demonstrated” due to lack of outcome data.
– OMB recommended OSEP develop a strategy for collecting outcome data
The Development of Outcome Statements
• 2004-early 2005: ECO generates discussion and gathers input on child and family outcomes
• Summer 2005: OSEP announces the child and family outcomes States must report on through their SPP/APRs
Early Childhood Outcomes Center4
Three Child Outcomes
Percent of children who demonstrate improved:– Positive social emotional skills (including
positive social relationships)– Acquisition and use of knowledge and skills
(including early language/ communication [and early literacy])
– Use of appropriate behaviors to meet their needs
Early Childhood Outcomes Center5
Key Concepts to Communicate
• Outcomes are functional– term has multiple definitions
• Each outcome is integrated across domains
• Challenge: Reporting language does not clearly define the outcomes
Early Childhood Outcomes Center6
Reporting Child Progress
• December 2005: States submit plans via SPP on how they will collect outcome data
• September 2006: OSEP finalizes the child outcome reporting categories
Early Childhood Outcomes Center7
Child Progress- the % of children who
a. did not improve functioning
b. improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers
c. improved functioning to a level nearer to same-aged peers but did not reach it
d. improved functioning to reach a level comparable to same-aged peers
e. maintained functioning at a level comparable to same-aged peers
Early Childhood Outcomes Center8
Key Concepts to Communicate
• Progress categories require 2 data points for each child,
• are based on growth trajectories,
• compare a child to him or herself over time, and also compare each child to age expectations
Early Childhood Outcomes Center9
Early Childhood Outcomes Center
10
Entry Exit
States Submit Data to OSEP in their Performance Reports
• Feb 2007: “status at entry” data
• Feb 2008: child progress data for children who exited 7/1/06 through 6/30/07
• Feb 2009: child progress data for children who exited 7/1/07 through 6/30/08
Early Childhood Outcomes Center11
State Approaches to Measuring Child Outcomes
Approach
Part C
(56 states/jur)
Preschool
(59 states/jur)
One tool statewide 7/56 (13%) 9/59 (15%)
Publishers’ online analysis
3/56 (5%) 6/59 (10%)
COSF 7 pt. scale 41/56 (73%) 38/59 (64%)
Other 5/56 (9%) 7/59 (10%)
Number of Children Included in Feb ‘09 SPP/APR Data
Part C (56)
Range: 5-6452
<30 = 3
30-99= 10
100-499= 25
500-999= 6
1000+ = 12
Preschool (59)
Range: 3-10157
<30 = 3
30-99 = 9
100-499= 14
500-999= 10
1000+ = 22
Key Concepts to Communicate
• Meaningfulness of the progress categories
• Establishing expectations for how the percentages “should” look
• Exploring relationships between state eligibility definitions and expectations for outcomes
Early Childhood Outcomes Center18
Setting Targets for Improving Child Outcomes
• 2008 and 2009: State input into summary statements to be the basis of target setting
• Currently: summary statements have been out for public comment, and are in draft form in SPP Measurement tables
• February 2010: Baseline data and target setting on summary statements
Summary Statement Data
• Required Summary Statement 1:
Of those children who entered or exited the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.
• Required Summary Statement 2:
The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.
20
Key Concepts to Communicate
• The meaning of the summary statements
• Ensuring that data is clean before making program changes based on it
• Choosing/ developing strategies that will improve child outcomes
Early Childhood Outcomes Center21
So where are the states now?
States and Outcomes Measurement
• Many states have embraced the ideas of measuring outcomes– Want information for own purposes– Program improvement
• The power of GOOD outcomes data is widely recognized
States and Outcomes Measurement
• States have invested resources and time in building an outcomes measurement system.– Involving stakeholders– Identifying an approach– Training providers in approach– Building a data system
And collecting data!
States have made incredible progress
• States started at various times
• Have varying amounts of data
• Examining their data (and procedures)– The Quality Question??
This is VERY DIFFICULT work
Some of the Challenges
• Need for good assessment tools– Need for more providers skilled in assessment
• Need strong grounding in typical child development• Adequate training statewide in data collection
procedures– Understanding and commitment
• Resources for new data systems • Capability to analyze, report and interpret data
– Generating reports for local programs
• Resources in general
Next steps:
Putting it all together
Early Childhood Outcomes Center28
Making Good Use of Data: Making Good Use of Data: Some Necessary Ingredients Some Necessary Ingredients
• Set of good questions
• Data set with the required elements
• Analytic capability to analyze the data to address the question
• Commitment to use the information for program improvement
Examining and Tweaking the Service System
Plan (vision) Program characteristics
Child and family outcomes
Implement
Check(Collect and analyze data)
ReflectAre we where we
want to be?
Is there a problem?
Why is it happening?
What should be done?
Is it being done?
Is it working?
Early Childhood Outcomes Center30
Information Infrastructure: Information Infrastructure: Data Needed for Program ImprovementData Needed for Program Improvement
WHO SERVICES
COSTQUALITY
OUTCOMES
Early Childhood Outcomes Center31
Where is your state (or program) now?Where is your state (or program) now?
WHO SERVICES
OUTCOMES
What do you have?
How much is linked?
WHOSERVICES
COSTQUALITY
OUTCOMES
WHOSERVICES
COST
QUALITY
OUTCOMES
WHOSERVICES
COSTQUALITY
OUTCOMES
PreschoolPreschool
KindergarteKindergartenn
Grade 1Grade 1
Grade 2Grade 2
WHOSERVICES
COSTQUALITY
OUTCOMES
Tracking outcomes over time
An outcomes measurement system
is more than hardware, software, and data
Framework Development
• ECO is working with 7 partner states over the next two years to develop a framework for outcomes measurement.
• Framework provides the basis for a self assessment to allow states to track their own progress in building an outcomes measurement system.
Areas of DRAFT Outcomes Measurement System Framework
Data Collection Data Collection and and
TransmissionTransmission
Data Collection Data Collection and and
TransmissionTransmission AnalysisAnalysisAnalysisAnalysis Information Information SharingSharing
Information Information SharingSharing ActionActionActionAction
Vision for What Is to Be LearnedVision for What Is to Be LearnedVision for What Is to Be LearnedVision for What Is to Be Learned
EvaluationEvaluationEvaluationEvaluation
Data Collection and Data Collection and TransmissionTransmission
• State has a method capable of accurately capturing outcomes information
• State has effective procedures for professional development around data collection.
• State has effective procedures for moving data from the local to the state level
• Etc.
State Self Assessment
• Is each area of the outcomes measurement system implemented?
• Is the area implemented in a way that reflects high quality?
We’ve come a long way, BABY!
39
…and we still have a long way to go…
...but won’t it be great when we get there?
For more information
(and for a copy of this presentation)
www.the-eco-center.org