Early Childhood Outcomes Center1 The Fun Stuff: Using the Data.

Early Childhood Outcomes CenterEarly Childhood Outcomes Center 1

The Fun Stuff: The Fun Stuff:

Using the DataUsing the Data


Using data for program Using data for program improvement = EIAimprovement = EIA

Evidence

Inference

Action


Evidence

Evidence refers to the numbers, such as

“45% of children in category b”

The numbers are not debatable


Inference How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news?

Bad news? News we can’t interpret?

To reach an inference, sometimes we analyze data in other ways (ask for more evidence)


Inference Inference is debatable --

even reasonable people can reach different conclusions from the same set of numbers

Stakeholder involvement can be helpful in making sense of the evidence


Action

Given the inference from the numbers, what should be done?

Recommendations or action steps

Action can be debatable – and often is

Another role for stakeholders


What can we infer?What can we infer?

Poll results A: Candidate I.M. Good 51%, Candidate

R.U. Kidding 49% (+ or – 3%)

Poll results B: Candidate I.M. Good 56%, Candidate

R.U. Kidding 44% (+ or – 3%)


Program improvement: Program improvement: Where and howWhere and how

At the state level – TA, policy

At the regional or local level – supervision, guidance

Classroom level -- spend more time on certain aspects of the curriculum

Child level -- modify intervention


Key pointsKey points

Evidence refers to the numbers and the numbers by themselves are meaningless

Inference is attached by those who read (interpret) the numbers

You have the opportunity and obligation to attach meaning


E – I – A JeopardyE – I – A Jeopardy

COSF users unaware of the need

to answer the yes/no progress

question

90% of exit COSFs in Program B

missing a response to the yes/no

progress question

Revise COSF procedures to

emphasize completion of yes/no

progress question

Conduct staff development on using the 7-point

rating scale

75% of children in Program A received

entry ratings of 2

COSF users misunderstand the definition of points

on the 7-point scale

Currently used tools are not accurately

assessing children’s social emotional

skills

Invest resources in materials for

assessing social-emotional skills

45% of children reported in category

‘e’ for statewide progress data,

Outcome 1

$100

$200

$100

$300

$200

$300

$200

$100

$300


Use of DataUse of Data

ActivityActivity

Evidence-Inference-Action


Continuous Program ImprovementContinuous Program Improvement

Plan (vision) Program characteristics

Child and family outcomes

Implement

Check(Collect and analyze data)

ReflectAre we where we

want to be?


Tweaking the SystemTweaking the System



Implement



want to be?

Is there a problem?

Why is it happening?

What should be done?

Is it being done?

Is it working?


Continuous means…Continuous means…

…….the cycle never ends..the cycle never ends.


Outcome questions for program Outcome questions for program improvement, e.g.improvement, e.g.

Who has good outcomes = Do outcomes vary by

Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received? Family outcomes? Education level of parent?


Examples of process questionsExamples of process questions Are ALL services high quality?

Are ALL children and families receiving ALL the services they should in a timely manner?

Are ALL families being supported in being involved in their child’s program?

What are the barriers to high quality services?


Working AssumptionsWorking Assumptions

There are some high quality services and programs being provided across the state.

There are some children who are not getting the highest quality services.

If we can find ways to improve those services/programs, these children will experience better outcomes.


Numbers as a toolNumbers as a tool

Heard on the street “Why are we reducing children to a

number?”

So why do we need numbers?





+35,000


+700,000+700,000


Examining COSF data at one time pointExamining COSF data at one time point

One group - Frequency Distribution Tables Graphs

Comparing Groups Graphs Averages


Distribution of COSF Ratings in FallDistribution of COSF Ratings in Fall

Outcome 1 Rating

N %

7 350 70

6 110 22

5 20 4

4 8 1.6

3 6 1.2

2 4 .8

1 2 .4

We are using fake data

for illustration


Frequency on Outcome 1 - FallFrequency on Outcome 1 - Fall

0

50

100

150

200

250

300

350

400

1 2 3 4 5 6 7



0

50

100

150

200

1 2 3 4 5 6 7


Comparison of two classes - FallComparison of two classes - Fall

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 70%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7



0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7

Class 1

Class 2


Frequency on Outcome 1 – Class 1Frequency on Outcome 1 – Class 1

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7

Fall

Spring


Average Scores on Outcomes by Class – Average Scores on Outcomes by Class – Fall, 2008Fall, 2008

ClassSocial-

Emotional

Knowledge and

Skills

Action to Meet

Needs

1 4.5 4.6 4.7

2 5.3 5.2 4.7

3 4.9 4.9 4.9

4 6.4 5.9 6.6

5 5.3 4.3 4.9

6 3.8 2.9 3.9

All Classes 5.03 4.63 4.95



ClassSocial-

Emotional

Knowledge and

Skills

Action to Meet

Needs

1 4.5 4.6 4.7

2 5.3 5.2 4.7

3 4.9 4.9 4.9

4 6.4 5.9 6.6

5 5.3 4.3 4.9

6 3.8 2.9 3.9

All Classes 5.03 4.63 4.95



ClassSocial-

Emotional

Knowledge and

Skills

Action to Meet

Needs

1 4.5 4.6 4.7

2 5.3 5.2 4.7

3 4.9 4.9 4.9

4 6.4 5.9 6.6

5 5.3 4.3 4.9

6 3.8 2.9 3.9

All Classes 5.03 4.63 4.95


Looking at change over timeLooking at change over time

Extent of change on rating scale The OSEP categories

Developmental trajectories Maintaining Changing


Extent of change on rating scale: Extent of change on rating scale: Time 1 to Time 2Time 1 to Time 2

Outcome 1 Progress N %

Maintained age-expected functioning

350 70

Maintained same level function, but not

age-expected60 12

Gained 3 steps 10 2

Gained 2 steps 25 5

Gained 1 step 50 10

Dropped 1 step 4 .8

Dropped 2 steps 1 .2


13

Functioning


Entry

14


Entry Exit

15


Entry Exit

16


Key PointKey Point

The OSEP categories describe types of progress children can make between entry and exit

Two COSF ratings (entry and exit) are needed to calculate what OSEP category describes a child progress


How changes in ratings on the COSF How changes in ratings on the COSF correspond to reporting categories a - ecorrespond to reporting categories a - e

e. % of children e. % of children who who maintainmaintain functioning at a functioning at a level level comparable to comparable to same-aged same-aged peerspeers

Rated 6 or 7 at Rated 6 or 7 at entry; ANDentry; AND

Rated 6 or 7 at Rated 6 or 7 at exitexit


Entry Exit


Entry Exit


Entry Exit



d. % of children d. % of children who improve who improve functioning to functioning to reachreach a level a level comparable to comparable to same-aged peerssame-aged peers

Rated 5 or Rated 5 or lower at lower at entry; ANDentry; AND

Rated 6 or 7 Rated 6 or 7 at exitat exit


Entry Exit



c. % of children c. % of children who improved who improved functioning to a functioning to a level level nearernearer to to same aged same aged peers, but did peers, but did not reach itnot reach it

Rated higher at Rated higher at exit than entry; exit than entry; ANDAND

Rated 5 or Rated 5 or below at exitbelow at exit


Entry Exit


Entry Exit



b. % of children b. % of children who who improvedimproved functioning, but functioning, but not sufficient to not sufficient to move nearer to move nearer to same aged same aged peerspeers

Rated 5 or lower Rated 5 or lower at entry; ANDat entry; AND

Rated the same Rated the same or lower at exit; or lower at exit; ANDAND

““Yes” on the Yes” on the progress progress question (b)question (b)


Entry Exit


Entry Exit


Entry Exit


Entry Exit



a. % of children who a. % of children who did not improvedid not improve functioningfunctioning

Rated lower at exit Rated lower at exit than entry; ORthan entry; OR

Rated 1 at both Rated 1 at both entry and exit; entry and exit; ANDAND

Scored “No” on the Scored “No” on the progress question progress question (b)(b)


Entry Exit


Entry Exit


Federal Reporting CategoriesFederal Reporting Categories

ActivityActivity

COSF Ratings to Progress Categories

ECO CalculatorECO Calculator


ECO Calculator (Tutor)ECO Calculator (Tutor)

http://www.fpg.unc.edu/~eco/outcomes.cfm




OSEP progress categoriesOSEP progress categories

Looking at information across time

Reducing the information to fewer categories to allow easier comparisons

OSEP Categories2009 (%)

2010(%)

2011 (%)

Maintained Age Appro Trajec 23 22 24

Changed Traj – Age Appro 15 17 13

Changed Traj – Closer to Age App 32 34 37

Same Trajectory -Progress 28 25 25

Flat Trajectory – No Prog. 2 2 1


2010(%)

2011 (%)



TOTAL - Age Appropriate at Exit 38 39 37

OSEP CategoriesClass 1

(%)Class 2

(%)Class 3

(%)



TOTAL - Age Appropriate at Exit 38 39 37


2010(%)

2011 (%)


Changed Traj – Closer to Age App 32 34 37

TOTAL – Greater than Expected Progress 47 51 55


Working with dataWorking with data

Different levels of analysis are required for different levels of questions

Aggregation will work for you – but loses detail about individual children.

50 assessment items on 20 children in 5 classes in Fall and Spring 50 x 20 x 5 x 2 = 10,000 pieces of

information


Using assessment data at the classroom Using assessment data at the classroom levellevel

Looking at the data by child At a single point in time Over time

Looking at data for areas that cut across children At a single point in time Over time

Name

Items Related to Outcome 1

1

Plays well with others

2Cooperates with peers in

simple games

3

Stops for transition

cues

4

Takes directions well from

adults

5

Has at least one

close friend

Carlos A E E A A

Geeta NY NY NY NY NY

Eileen A A A A A

Ming E E E NY NY

Shaniqua E E E E A

Example: Item Results for 5 Imaginary Children

A=Accomplished; E= Emerging; NY= Not yet

Example: COSF Outcome Ratings for Class 3c by Child

Name

Outcome 1 Outcome 2 Outcome 3

Time 1

Time 2

Time 1

Time 2

Time 1

Time 2

Carlos5 6 3 4 5 5

Geeta1 2 2 2 3 4

Eileen7 7 7 7 7 7

Ming2 2 4 4 2 3

Shaniqua3 4 3 4 5 5

Example of an Aggregated Report for Program: Percentage of Children Scoring 5 or Higher on COSF by Class

Class

Outcome 1 Outcome 2 Outcome 3

Time 1 Time 2 Time 1 Time 2 Time 1 Time 2

1a 65 70 50 51 49 52

1b 55 53 62 61 87 88

2a 47 43 51 67 65 66

2b 76 84 78 85 78 83

3a 97 98 95 97 95 100

What do you see in these data?


Outcome questions for program Outcome questions for program improvement, e.g.improvement, e.g.

Who has good outcomes = Do outcomes vary by

Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received? Family outcomes? Education level of parent?


Looking at Data by RegionLooking at Data by Region

Class 1 Class 2 Class 3

45 47 23

Percentage of Children Who Changed Developmental Trajectories

After One Year of Service

Possible inference?


Looking at Data by Age at EntryLooking at Data by Age at Entry

36 to 40 months

41 to 44 months

45 to 49 months

34 42 46

Percentage of Children Who Changed Developmental Trajectories

After One Year of Service

Possible inference?


Take Home MessageTake Home Message

You will want to look at your data in lots of different ways

You will want to think about the possible inferences

You may need other information to decide among possible inferences

Act on what you have learned


Tweaking the SystemTweaking the System



Implement



want to be?

Is there a problem?

Why is it happening?

What should be done?

Is it being done?

Is it working?


How will/might these data be used?How will/might these data be used?

Federal level Overall funding decisions (accountability) Resource allocation (e.g., what kind of TA to fund?) Decisions about effectiveness of program in individual

states State level

Program effectiveness?? Program improvement??

Local level Program improvement??


Need for good dataNeed for good data

Encompasses all three levels: federal, state, local

Depends on how well local programs are implementing procedures


Many steps to ensuring quality dataMany steps to ensuring quality data

BeforeGood data collection/Training

Good data system and data entry procedures

During

Ongoing supervision of implementation

Feedback to implementers

Refresher training

AfterReview of COSF records

Data analyses for validity checks


Take Home MessageTake Home Message

If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.

Inference = Data not yet valid

Action = Continue to improve data collection and quality assurance


Data ExplorationData Exploration

Examine the data to look for inconsistencies

If and when you find something strange, look for some other data you have that might help explain it. Is the variation caused by something other than bad data?


Obtaining good dataObtaining good data

Focus on addressing the threats to good data Local providers do not understand the

procedures Local providers do not follow the

procedures And others…..

Identify and address the threats


How far along is Illinois?How far along is Illinois?

Quality of Child Outcomes Data

Time


Keeping our eye on the prize:

High quality services for children and

families that will lead to good outcomes.


For more information….

www.the-eco-center.org

Early Childhood Outcomes Center1 The Fun Stuff: Using the Data.

Documents

data slide

good outcomes

family outcomes

better outcomes

action slide

debatable slide

intervention slide

illustration slide