Accountability: Getting it Right in Early Childhood Kathy Hebbeler Early Childhood Outcomes Center at SRI International Presented at the Measuring Child.

Accountability: Accountability: Getting it Right Getting it Right

in Early Childhoodin Early Childhood

Kathy HebbelerKathy HebbelerEarly Childhood Outcomes CenterEarly Childhood Outcomes Center

at SRI Internationalat SRI International

Presented at the Measuring Child and Family Outcomes Meeting, Baltimore 2008

Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2

Where are you?


Today we hope to…Today we hope to…

Begin a conversation about what an ideal EC accountability system would look like

Describe some recent developments in EC Accountability

Support you as state leader in shaping the EC accountability system in your state.


What is happening out there?What is happening out there?

Incredible expansion of EC programming at the state level

Accountability is not going away How best to do accountability is still a

topic for hot conversation in early childhood

Essential that persons knowledgeable about IDEA programs be part of those conversations


For what do we want to be held For what do we want to be held accountable?accountable?

a) High quality services provided through 619 and Part C

b) Good outcomes for children with special needs and their families

c) Both


And who are “we?”And who are “we?”

a) Administrators and others associated with Part C and Part B 619 programs?

b) Advocates concerned about how young children in our society are faring?

c) Both


Good Accountability requiresGood Accountability requires

Intentionality Being proactive Having a vision

So what is your state’s vision for your accountability system?


Good Accountability requires..Good Accountability requires..

Seeing accountability as a

system…. …A group of interacting,

interrelated, or interdependent elements forming a complex whole


EC System (or not)EC System (or not)

State Pre-K

Public Health home visiting

Child Care

Part C

619Early Head Start

Head Start


System Under ConstructionSystem Under Construction

State Pre-KChild CarePart C

Head Start

TransitionAccountability

619


Components of IDEA Service SystemComponents of IDEA Service System

Service Delivery Child Find System entry – eligibility determination Service provision Transition Etc.

Supports to Service Delivery Policy, legislation, regulation Professional Development Accountability Etc.


The Accountability System is a component within the Service System

System within a system


Two Recent ReportsTwo Recent Reports


State Early Childhood Accountability and State Early Childhood Accountability and Improvement SystemImprovement System

A system of standards-based assessments of

(a) children’s development and learning and

(b) program quality,

designed to inform state policy decisions, investments, and improvement efforts for early education programs for three- and four-year-old children,

linked to a continuum of kindergarten through third grade standards, curriculum, assessments, and program improvement efforts.

National Early Childhood Accountability Task Force




(b) program quality,

designed to inform state policy decisions, investments, and improvement efforts for early education programs for three- and four-year-old children,

linked to a continuum of kindergarten through third grade standards, curriculum, assessments, and program improvement efforts.

State Early Childhood Accountability and Improvement State Early Childhood Accountability and Improvement SystemSystem

National Early Childhood Accountability Task ForceChi

ld O

utco

mes

Linkage

Purpose

Program information


Three Primary Building BlocksThree Primary Building Blocks

System Infrastructure Assessment/Program Improvement

Approaches Steps toward Coherent PreK-Grade 3

Accountability Efforts



Infrastructure Infrastructure

….An underlying base or foundation especially for an organization or system

….The basic facilities, services, and installations needed for the functioning of a community or society



Four Parts of the EC Accountability Four Parts of the EC Accountability InfrastructureInfrastructure

Early learning and program quality standards

Program rating and improvement system

Professional development system Data management and reporting

systemNational Early Childhood Accountability Task Force


Assessment: Importance of Purpose Assessment: Importance of Purpose

Make purpose of assessment explicit and public

Match the assessment strategy to the purpose

Administrators, teachers, and assessors should be able to articulate the purpose of an assessment

NRC Report - Early Childhood Assessment: Why, What, and How?


Assessment system is part of EC larger Assessment system is part of EC larger systemsystem

Standards Assessments Reporting (data base) Professional development related to standards and

assessment Procedures to assess opportunity to learn (Program

quality) Fair assessment for all children Sufficient resources to implement Ongoing monitoring and evaluation

NRC Report - Early Childhood Assessment: Why, What, and How?


Relationship between these two reportsRelationship between these two reports

Task Force Accountability Report – Focus: EC Accountability

Assessment is a component of an accountability system

NRC Report on Assessment Focus: EC Assessment

Accountability is one purpose for assessment


Components of an Accountability SystemComponents of an Accountability System

Vision (related to Vision for Service System)

Purpose


Articulating purpose for state Articulating purpose for state Accountability SystemAccountability System

Probably goes beyond “Reporting data for the SPP/APR”

Needs to be more specific than “accountability and program improvement”

Note: Purpose can vary with level; OSEP can have different purposes from state’s purpose(s) as can local programs.


Possible purposesPossible purposes

Demonstrate effectiveness of program for policy-makers (governor, legislature)

Identify local programs in need of improvement Determine whether services are equally effective for

different sub-groups, e.g., Children with health problems Low income families, etc.

Determine whether services are equally effective across different outcome areas.

_______________________________________

Check all that apply


Communication is keyCommunication is key

Purpose(s) needs to be Clearly articulated In writing Readily available Widely disseminated

No searching. No secrets. No suspicions.


How to Achieve Purpose(s)?How to Achieve Purpose(s)?

1. Build evidence base so state can reach valid conclusions about service system.

2. Use evidence base to accomplish stated purposes.


Build the evidence baseBuild the evidence base

FMA* What does state want to know about

what?

*Findings, Meaning, Action


Evidence about what?Evidence about what?

Accountability Task Force:



(b) program quality

Note: This assumes the service system has standards for children’s development and learning and standards for program quality so they can be assessed


Standards for Children’s Learning and Standards for Children’s Learning and Development: Early Learning GuidelinesDevelopment: Early Learning Guidelines

Knowledge, skills, habits, attitudes, dispositions children are expected to acquire by a given point in time

Almost every state has them for 3-5 year olds

Many states already have them for 0-3; others with them under development.



Early Learning GuidelinesEarly Learning Guidelines

Developed for multiple purpose Considerable variations in content,

format, specificity across states Stand as public statements of what we

want children to do at various ages Public policy statement of

developmental expectations




What we know (WWK)What we know (WWK)

To paraphrase Thomas Jefferson, “We hold these truths to be evidence-based….


WWK: Children’s development proceeds WWK: Children’s development proceeds according to a general timelineaccording to a general timeline

Competencies are acquired within an age range; developmental milestones

Some skills precede other skills Developmental timeline is the basis

for screening and assessment tools in early childhood; also provides foundation for ELGs


WWK: Current developmental status WWK: Current developmental status predicts future developmental statuspredicts future developmental status

Current functioning is a statement about where a child is

Also, a prediction or odds for where the child will be in the future.

Basis for developmental trajectories Goal of intervention is to change the

odds.

Illustration of 5 Possible Develomental Trajectories

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Maintained functioning comparable to age peersAchieved functioning comparable to age peersMoved nearer functioning comparable to age peersMade progress; no change in trajectoryDid not make progress


WWK: There is a lot of variation in children’s WWK: There is a lot of variation in children’s development/learning but some children’s development/learning but some children’s development/learning is problematicdevelopment/learning is problematic

Some is “just variation” Some is troublesome – because it bodes ill

for the child’s future (bad odds) Critical that we can distinguish Policy response:

Early intervention and early childhood special education

State criteria for delay – the point at which the delay is so substantial, the child is not likely to “outgrow” it. Eligibility for IDEA services.


Thinking About Age ExpectationsThinking About Age Expectations

Age-expectedskills and behavior

Movement away from age-expected

Movement toward age-expected

Inequality at Inequality at

ECLS-K data from Inequality at the Starting Gate

inequalityinequality

ECLS-K data from Inequality at the Starting Gate



Policy response: Head Start State Pre-K Child Care Quality Standards

Goal: Alter the trajectory. Change the odds



Policy response: Early learning guidelines

[Hopefully] Grounded in what we know about development, ELGs lay out developmental expectations

Provide guidance to keep children “on track” or get them back on track so they arrive at kindergarten with a certain set of skills


Understanding the difference between Understanding the difference between odds and a sure thingodds and a sure thing

Need to understand how to interpret data on children’s current functioning

Not wise to obsess on slight degrees of variation

Not wise to ignore substantial deviation (“grow out of it”)

A child “off track” is a prediction begging to be proven wrong – if effective intervention is provided (WWK: Early intervention works.)


Application of ELGs to children with Application of ELGs to children with special needsspecial needs

Do ELGs apply? Are they useful? At a individual child or classroom level? At a state level?

They are statements of what is expected at given ages

Parallels with the application of standards for children with disabilities in K-12?

Have fought for students’ rights to be held (and compared) to the same standards as all students

Standards provide access to higher expectations for students with disabilities

Could ELGs be a common currency in moving toward unified EC system?


Application of ELGs to children with Application of ELGs to children with special needsspecial needs

DEC Recommended Practice: A41. Professionals monitor child progress based on past performance as the referent rather than on group norms.

This RP is speaking to norm-referenced tests. Implication is that the only thing we want to look at

is child compared to him’/herself. ELGs are not group norms – but they are group

standards. Can ELGs provide directions to Part C and 619

programs for age expectations?


Lessons ECO has learned Lessons ECO has learned

EI and 619 providers need to know more about typical child development And so do child care providers, etc…

Some lack of clarity over how to determine age expectations for OSEP reporting purposes

Provider discomfort over how to talk to parents about current level of functioning when it is low


PA’s red flags “…help adults gain a

sense of whether additional screening or evaluation should be sought if a particular skill has not been acquired within a specific developmental age span.”

Help adults watch for possible delays.


What kind of evidence about children’s What kind of evidence about children’s learning and development?learning and development?

Task Force Recommendation: Measure progress and status.

OSEP reporting does both! Checks on status at 3 years and 5 years. Looks at progress over time in program.


Evidence about what?Evidence about what?

Accountability Task Force:



(b) program quality

Recommended expansion: Include family outcomes.


What do we want to know about What do we want to know about programs?programs?

Both ATF and NRC reports speak to looking at program quality

Possible addition: Reframe to be program characteristics and program quality Program characteristics:

Hours, months of operation Program model Services offered Curriculum

Program quality measures



Important difference in “program” for C/619 compared to general EC Some program measures apply to a

group of children (e.g., measures of classroom quality)

Some are individualized (e.g., receipt of speech therapy)

Important implications for measurement



What is the child’s program? Services Minutes of services Location of service provision

What is the quality of the child’s program? How should we assess quality, especially

of the individualized component?


Program quality measuresProgram quality measures

Examples: Early Childhood Rating Scale (ECERS), Early Language and Literacy Classroom Observation (ELLCO), Classroom Assessment Scoring System (CLASS)

See NRC report for a comprehensive review of program quality tools


Evidence about what? Possible Evidence about what? Possible Modifications to ATF RecommendationsModifications to ATF Recommendations

Child and family outcomes Program (Group, individual)

Characteristics Quality

Infrastructure supporting the program Legislation policies Professional development, etc.

System for Producing Good Child and System for Producing Good Child and Family OutcomesFamily Outcomes

High quality services and supports for children 0-5 and their families

Good outcomes for children and families

Prof’l Development•Preservice•Inservice

System for Producing Good Child and System for Producing Good Child and Family OutcomesFamily Outcomes

Good Federal policies and programs

Good State policies and programs

High quality services and supports for children 0-5 and their families

Good outcomes for children and families

Good Local policies and programs

Adequate funding

Strong Leadership


Ecological Model of Child DevelopmentEcological Model of Child Development

Child

Family


Ecological Model of Child DevelopmentEcological Model of Child Development

Child

Family

Community


Ecological Model of Program Ecological Model of Program ImplementationImplementation

Multiple forces impact how programs are implemented

Important to know how your programs look

Also, important to know what leads them to look like this

Forces acting on programs provide levers for program improvement


Concluding CaveatConcluding Caveat

NEVER ever reach a conclusion about program effectiveness based on child outcome data alone

Best practice (at all levels of decision-making): Multiple sources of information

Conclusion: Accountability system needs evidence about outcomes AND evidence about program




Purpose Evidence Base


How to get the evidence How to get the evidence

1. Good measurement procedures = procedures capable of producing high quality date

Good system design

2. Fidelity of implementation = procedures are implemented as intended

Good quality assurance




Purpose Evidence Base Use of evidence to accomplish

purposes


Use of evidence* Use of evidence*

Reaching valid conclusions (M) Taking effective action (A)

*Addressing the purposes


Taking the fear out of accountabilityTaking the fear out of accountability

What are we afraid of? Findings/Evidence Meaning/Conclusions Action

Continuum of actions What are the options on the continuum?


Accountability NightmareAccountability Nightmare

Meaning – reaching the wrong conclusion

Action – leading to actions that will ultimately harm children and families


Avoiding the nightmareAvoiding the nightmare

Type of wrong conclusion False negative –program is ineffective

when it is effective Overall For some types of children In some areas

False positive – program is effective when it isn’t


Alignment of Conclusion and RealityAlignment of Conclusion and Reality

Effective Ineffective

Effective Correct!False

Positive

IneffectiveFalse

NegativeCorrect!

Reality

Conclusion


Avoiding the nightmareAvoiding the nightmare

Good evidence Credible Multiple pieces of evidence

Evidence about child outcomes Evidence about program factors (quality)*

*See Accountability Task Force


Need for outcomes AND program dataNeed for outcomes AND program data

Avoid both false negatives and positives by looking at multiple pieces of evidence Triangulation Do multiple sources of evidence point to

the same conclusion? The outcomes data; the program data;

data on who is being served, etc.


Avoiding the nightmare: Avoiding the nightmare: Solid Infrastructure for an Accountability Solid Infrastructure for an Accountability SystemSystem

Clear vision Clear articulation of purpose and

public statement as to how the data will be used

Credible evidence (drill down here…)


Clear statement of possible future actionsClear statement of possible future actions

What will be the consequences of a negative finding?

Sanctions?? The ultimate worst nightmare is the

prospect of a false negative combined with sanctions

Never consider sanctions based on child outcomes data alone (or any single piece of evidence)


Clear statement of possible actionsClear statement of possible actions

Supports to improve weak programs or weak program areas Site visits Individualized technical assistance Professional development plan Etc.

Note: If the wrong program is identified (false negative) and provided resources, no harm comes to children and families in that program but it is a waste of resources.


Clear statement of possible actionsClear statement of possible actions

False positives Sort of where we are now: programs,

investments assumed effective ..but some of them probably aren’t

Consequences Short changes children and families Some children and families will not

experience the kinds of outcomes they could have with a quality program.


Sidebar thoughtSidebar thought

Why are we more worried about false negatives than false positives?

It would be damaging to children and families to defund an effective program but it also is damaging to have children and families receiving ineffective services….


Sidebar thought #2Sidebar thought #2

What about comparing states [programs]?

What kinds of conclusions will OSEP reach?

What are the actions/consequences that will follow?

OSEP has no intention of comparing states on outcomes


What about comparing?What about comparing?

Can’t afford to be naïve Once data are out there, anyone can make

comparisons (state to state; within state, local to local)

Comparing is not the problem; invalid conclusions (especially false negatives) are the problem

Using comparisons to identify weak programs is what accountability and program improvement is supposed to do


Best protection against false negatives Best protection against false negatives (and false positives)(and false positives)

Good evidence base Multiple sources of information on:

Children and families Programs Infrastructure

Credible, well implemented measurement strategies


Putting Good Accountability to WorkPutting Good Accountability to Work

1. Build the infrastructure for producing a body of evidence that will allow valid conclusions

2. Compile the evidence and carefully reach conclusions

3. Act on those conclusions to build a better system of services

4. Repeat 2 and 3 indefinitely.


Good EC SystemsGood EC Systems

Build on what we know (WWK) About children’s learning and development

Multiple influences on development About adult learning About innovation, implementation Etc. (this is another way of saying they are

evidenced-based) To provide high quality services and

supports


Accountability Task Force Accountability Task Force RecommendationRecommendation

States should develop a single unified and coherent system of standards, assessments, data and professional development efforts across early childhood categorical programs and funding streams.

**Alignment****Systems Integration**


Turning an EC Non-System into a SystemTurning an EC Non-System into a System

What resources exist in your state to promote good outcomes for children birth to 5?

Which of these resources work together? How?

Share a vision of common outcomes? Build a seamless system for families? Joint professional development? Shared vision for transition? Unified accountability system?


Producing Good Outcomes for Young Producing Good Outcomes for Young Children with DisabilitiesChildren with Disabilities

Proposition: Good IDEA services are not enough to effectively change the odds for some(?)/many(?) young children with special needs.

Need the entire EC System to be functioning effectively.



Need more forces at work to change the odds and help children meet developmental expectations.

Need good child care (and an accountability system)

Need good PreK and Head Start programs



Need the EC services to work together

Need an accountability system that can look at the entire EC system


For what do we want to be held For what do we want to be held accountable?accountable?

a) High quality services provided through 619 and Part C

b) Good outcomes for children with special needs and their families

c) Both


And who are “we?”And who are “we?”

a) Administrators and others associated with Part C and Part B 619 programs?

b) Advocates concerned about how young children in our society are faring?

c) Both


ReferencesReferences

Task Force on Accountability (2007): Taking Stock: Assessing and Improving Early Childhood Learning and Program Quality

http://www.pewtrusts.org/our_work_detail.aspx?id=102

National Research Council; Snow & Van Hemel (2008): Early Childhood Assessment: What, Why, and How?

http://www.nap.edu/catalog.php?record_id=12446


ReferencesReferences

Web site on state Early Learning Guidelines http://www.ccsso.org/content/PDFs/State_ECstandardsMA

TRIX11.27.07.pdf

Research on State’s ELGs –Search on “Catherine Scott Little” on Google Scholar

Lee, V. E., & Burkam, D. T. (2002). Inequality at the starting gate: Social background differences in achievement as children begin school. Washington, DC: Economic Policy Institute.

Accountability: Getting it Right in Early Childhood Kathy Hebbeler Early Childhood Outcomes Center at SRI International Presented at the Measuring Child.

Documents

system slide

early childhood essential

early head

service system system

improvement system

system of standards

ideal ec accountability

transition accountability