Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,

Approaches to Measuring Approaches to Measuring Child OutcomesChild Outcomes

Approaches to Measuring Approaches to Measuring Child OutcomesChild Outcomes

Kathy HebbelerECO at SRI International

Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes, Albuquerque, NM April 2006

2

What is happeningWhat is happening

Outcomes measurement is difficult and very complex

No one group has all the answers – or even most of the answers

There are some exciting things going on around the country

3

Purpose of the meetingPurpose of the meeting

Share our challenges and what we have learned so far

Contribute to the collective knowledge base

Advance the discussion incrementally moving closer to producing outcomes data

*** for the ultimate good of children and families ***

4

Critical eventsCritical events Spring 2005 – ECO submitted

recommendations to OSEP on what should be collected with regard to child and family outcomes

Summer 2005 – OSEP released the reporting requirements

December 2005 – States submitted their plans for outcome data collection in their State Performance Plan

Spring 2006 – States are collecting data

5

The Number One Question*:The Number One Question*:

What are other states doing?

Have other states done X?

*In two forms

6

Purpose: The Overriding Purpose: The Overriding QuestionQuestion

Why is a state collecting data on child outcomes?

Context, resources, values, etc. enter into the answer

7

To respond to federal

reporting requirements

To meet provider/teacher, local and/or state need for outcome

information and

to respond to federal reporting

requirements

Purpose

WHY?

Context & Context & Values Drive Values Drive

DecisionsDecisions

Resources?

Stakeholder input?

Burden on locals?

Standardized assessment?

Authentic assessment?

Interagency issues?

Local control?

Minimize change?Early learning guidelines?

Multiple sources of information?

Policymakers want data?

Other early childhood initiatives?

9

Who is included in the outcomes Who is included in the outcomes system?system?

Pt. C system; Pt. B system Some blending of C and B

Same assessment Data sharing Data linking

Early Childhood System that includes C and B

10

How does the state get data on How does the state get data on outcomes?outcomes?

Who provides? What assessments are used? How often is data collected? When is data collected? (When is it

reported?) Dealing with multiple sources? Dealing with different assessments?

How Outcomes Data Get to the State AgencyHow Outcomes Data Get to the State Agency

12

Analysis of SPPsAnalysis of SPPs

Analyses are based on SPP reports submitted in December 2005

Pt. C N=56; Pt B N= 58 Limitations

Variation in level of detail provided Landscape keeps changing

Analysis done by Lynne Kahn and staff at UNC/FPG

13

““Camera” Issue: Capturing Child Camera” Issue: Capturing Child FunctioningFunctioning

What are the sources of the information on child functioning?

What kind of assessment tools are states planning to use?

14

Capturing Child Functioning: Capturing Child Functioning: How many sources?How many sources?

Multiple sources Pt. C: 50% (28 states) Pt. B: 16% (16 states)

One data source Pt C: 39% (22 states)

Assessment instrument (21 states) Pt B: 55% (32 states)

Assessment instrument (31 states)

15

Issues RaisedIssues Raised Data needs to reflect a child’s

functioning in each broad outcome area Functional outcomes summarize each

child’s current functioning across settings and situations

Best practice for assessing young children recommends the use of multiple measures

Will single sources (= assessment tool) Will single sources (= assessment tool) produce valid data on functional produce valid data on functional outcomes? How good is the camera?outcomes? How good is the camera?

16

The Child OutcomesThe Child Outcomes

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to meet their needs

17

Part C Outcomes Data SourcesPart C Outcomes Data Sources

Data Source # %

Formal assessment instruments

45 80%

Parent report 25 45%

Observation 14 25%

Clinical opinion 10 18%

IFSP goals & objectives 6 11%

Record review 4 7%

Not reported 6 11%

18

Preschool Outcomes Data SourcesPreschool Outcomes Data Sources

Data Source # %

Formal assessment instruments

45 80%

Observation 12 21%

Parent report 11 19%

Teacher/provider report 8 14%

IEP goals & objectives 1 2%

Clinical opinion 1 2%

Not reported 10 17%

19

Role of FamiliesRole of Families Impossible to understand how a child is

functioning across a variety of everyday settings and situations without family input

Options Incorporated into the assessment tool Collected through a parent-completed tool Incorporated into a summary rating

Issue: How is information from families Issue: How is information from families being included?being included?

20

Capturing Child Functioning: Capturing Child Functioning: Approaches to identifying Approaches to identifying assessment toolsassessment tools

One assessment selected by state

List of assessments developed by state; programs pick

Programs can use whatever they have been using

21

Capturing Child Functioning: Capturing Child Functioning: Assessment Tools Being UsedAssessment Tools Being Used

Part C – 20 different assessment tools identified 3 states using state developed tool

Part B – 43 different assessment tools identified 7 states using state-developed tool

22

Commonly Reported Commonly Reported Assessment Instruments: Part CAssessment Instruments: Part C

Of 28 states who listed specific assessment instruments: HELP - 15 states BDI/BDI2 - 13 states AEPS - 11 states Creative Curriculum - 6 states ELAP- 6 states

Not reported – 30 states Not yet determined - 23 states

23

Commonly Reported Commonly Reported Assessment Instruments: PreschoolAssessment Instruments: Preschool

Of 31 states who listed specific assessment instruments: BDI/BDI2 - 9 states Creative Curriculum - 8 states Brigance- 7 states High Scope COR - 6 states AEPS - 5 states State developed assessments - 7 states

Not reported - 27 states Not yet determined – 21 states

24

Capturing Child Functioning: Capturing Child Functioning: Combining Information from Multiple Combining Information from Multiple SourcesSources

Part C : Using ECO Summary Form – 52% (29

states) Developing own summary tools – 7%

(4 states) Part B:

Using ECO Summary Form – 29% (17 states)

Developing own summary tools – 10% (6 states)

25

Capturing Child Functioning: Capturing Child Functioning: TimingTiming

When and how often outcome information is being collected is related to why state is collecting data

What assessment is also related to why

OSEP requirement is entry and exit

26

When will data be “collected”?When will data be “collected”?

Aligned around the naturally occurring data review points in programs

“Collected” may mean Data reviewed/summarized to determine

a functional level for each of the outcomes

Summary rating or other data reported to state or OSEP

Some states did not report anything beside at entry and exit; (C – 28 states; B - 15 states)

27

When entry data will be collectedWhen entry data will be collected(three general patterns)(three general patterns)

Referral Eligibility Initial IFSP- e.g. goals, services, settings

IFSP 6 month review- intervention planning

Around eligibility (based on evaluation data)

Initial IFSP (based on eval and assessment data)

After services begin (based on eval, assess, and/or ongoing progress monitoring data

28

Part C examples of when data Part C examples of when data will be collectedwill be collected

W/in 45 days of referral- 7 states W/in 1 month of IFSP- 2 states W/in 6 months of enrollment- 1 At initial IFSP and 6 month and

annual reviews- 21 W/in 2 months, 45 days, 3 months,

6 months of exit

29

Preschool examples of when Preschool examples of when data will be collecteddata will be collected

Initial evaluations/eligibility - 7 states Initial IEP development -4 states Annual IEP reviews - 11 states Time periods prescribed by curriculum

referenced tools (2 or 3 times a year)- 8 states

Annually at the end of the school year- 6 states

30

Which Children Will Be Included: Which Children Will Be Included: Part C Part C

All children – 40 states After a pilot or phase in period- 16 states

Sampling – 7 states 1 sampling at exit (all children will have

entry data) 1 sampling at entry (will only collect entry

and exit data on children in sample) Other 5 - could not tell from SPP

Not reported or undecided - 9 states

31

Which Children will be Included: Which Children will be Included: PreschoolPreschool

All children – 42 states After a pilot or phase in period- 15 states

Sampling - 8 states 3 will collect data on ALL children, but select

a sample to report to OSEP 1 sampling at entry (will only collect entry

and exit data on children in sample) Other 4 - could not tell from SPP

Not reported or undecided- 8 states

32

Collaboration between C and B Collaboration between C and B

25 states reported in Part C SPP collaborating with Part B on outcomes

21 states reported in Part B SPP collaborating with Part C on outcomes

33

Collaboration with Other Early Collaboration with Other Early Childhood InitiativesChildhood Initiatives

Collaborate with or align outcome efforts with broader early childhood accountability initiatives in their state Part C: 3 states Part B: 18 states

Issue: What are the outcomes being Issue: What are the outcomes being assessed in the broader initiatives?assessed in the broader initiatives?

34

Role of the Early Learning Role of the Early Learning GuidelinesGuidelines

May change or add to the outcomes questions

Are children meeting the ELGs?Are children meeting the ELGs? May mean mapping the ELGs to

the 3 OSEP outcomes Aligning with ELGs:

Part C – 8 states; Part B – 18 states

How Outcomes Data Get to the State AgencyHow Outcomes Data Get to the State Agency

36

Transfer Issues: How does Transfer Issues: How does information move?information move?

In what form? At what level of

detail? With what level of

identification?

37

In what form?In what form?

Online In an electronic

file On paper

38

At what level of detail?At what level of detail?

Child Level Data Item level data on the child (from an online

assessment system) Scores on assessment tool ECO Summary Rating OSEP Categories (a, b, c) Other?

Aggregated Data Scores, Rating, OSEP categories, etc.

39

With what level of With what level of identification?identification?

Only relevant for child-level data Can state link outcome data be linked to

other information though an ID? Does it enter the system already linked?

Linkage to other data has major Linkage to other data has major implications for analysis and implications for analysis and questions state will be able to questions state will be able to answeranswer

40

Training Training

Focused on various topics Training in assessment tools Training in use of the ECO Summary Form

Various approaches Various levels of investment ECO is developing materials and

compiling training materials for web site (including materials designed for parents)

Contact NECTAC or ECO for help

41

ConclusionsConclusions

States are building many different kinds of outcomes measurement systems

Features of the system reflect the contexts and values of the state

We know some things about what states are doing but the landscape keeps changing