Critical Markers of High Quality Child Outcomes Data ECO Advisory Board March, 2012.

Critical Markers of High Quality Child Outcomes Data

ECO Advisory BoardMarch, 2012

Topics

• Why identify critical markers of high quality data?

• Anticipated use• 3 areas of focus• Draft critical markers• Where these might be

discussed in the APR• Reactions?

2Early Childhood Outcomes Center

Looking for…

Proposed markers that suggest a state’s EC outcomes systems may produce high quality data

Use markers to:

- systematically track over time - track internally (within states)

- to produce a national picture


Anticipated Use

• Tracking state progress (based on APR info)• Compare data from each state to a series of

critical markers for summaries– X % of states met standard on each marker nationally– X% of states met standard on 7 out of 10 markers

• NOT a state by state report card• Share detailed info with each state, upon

request


3 Areas

• Completeness of data


Completeness of data

• Missing data concepts– Are whole forms missing? (Have data

from all kids expect to have data from)

– Are the forms that you have complete?


3 Areas


• Accuracy of data


Accuracy of Data

• Do the data reflect the “true” performance of children in the program– Unknowable, so….

• Are the patterns in data what you would expect to see if the data were accurate?

• Do they make sense? Or, are there red flags that raise more questions?


3 Areas


• Accuracy of data

• State efforts related to tracking quality of data


Criteria for Selection of Markers

• Important – necessary for or indicative of

high quality data• Accessible to ECO

– Information to determine presence or absence of marker is available


Draft Markers: Completeness

• State calculates and publicly reports number of children missing outcomes data

• Percent of missing outcomes data is less than 5%. – Is this reasonable?

• Percent of missing data by proxy calculation is less than. – 40% of exiters for Part C– 20% of child count for 619


Part C percent of exiters 2009-10


Draft Markers: Accuracy/Patterning

• State % in a is not overly high (GT 5%)• State % in b is not overly low (LT 5%) or high (GT 50%)• State % in c is not overly low (LT 5%) or high (GT 50%)• State % in d is not overly low (LT 5%) or high (GT 50%)• State % in e is not overly low (LT 5%) or high (GT 65%)

13Early Childhood OutcGTomes Center

Knowledge and SkillsPart B 619 proportion of children that make no

progress (progress category a)


Knowledge and SkillsPart B 619 proportion of children who improved functioning but not sufficiently to move nearer to functioning comparable to same age

peers(progress category b)


Knowledge and SkillsPart B 619 proportion of children who improved functioning to a

level nearer to same age peers but did not reach it(progress category c)


Knowledge and SkillsPart B 619 proportion of children who improved functioning to

reach a level comparable to same aged peers(progress category d)


Knowledge and SkillsPart B 619 proportion of children who maintained

functioning at a level comparable to same aged peers(progress category e)


Percent of states meeting the Critical Markers for 2009 - 10


Accuracy of data Completeness of data

Both

Part C 66 59 47

Part B 619 63 66 41

Draft Markers: State Quality Review of Data

• State conducts data quality checks• State’s own analyses provide evidence of

high quality data



Completeness/ Missing Data

Accuracy/ Patterns

Location in Suggested APR Templatehttp://www.fpg.unc.edu/~eco/pages/fed_req.cfm#ECOSuggestedFormats


State Quality Review of Data

Location in Suggested APR Templatehttp://www.fpg.unc.edu/~eco/pages/fed_req.cfm#ECOSuggestedFormats

Questions for you

• Are these markers important?– Related to high quality data– Things you value and might track or are already tracking

them?– Would tracking them help you improve your system?

• Is there anything else you can think of as a good marker of high quality data


Summary – Draft Critical Markers

• State calculates and publicly reports number of children missing outcomes data

• Percent of missing outcomes data is less than 5%• Percent of missing data by proxy calculation is less than X • State % in a is not overly high (GT 5%)• State % in b is not overly low (LT 5%) or high (GT 50%)• State % in c is not overly low (LT 5%) or high (GT 50%)• State % in d is not overly low (LT 5%) or high (GT 50%)• State % in e is not overly low (LT 5%) or high (GT 65%) • State conducts data quality checks• State’s own analyses provide evidence of high quality data

24

Comments, Reactions, Questions?

25

Critical Markers of High Quality Child Outcomes Data ECO Advisory Board March, 2012.

Documents

infocompare data

tracking quality of

percent of missing data

state tracking

high quality data use

completeness state

high gt

data similar minimum