Measuring Child and Family Outcomes Session One Linking Assessment to Functional Outcomes and Accountability Anne Brager, MS, RN Program Supervisor Frederick.

Measuring Child and Family Outcomes

Session OneLinking Assessment to

Functional Outcomes and Accountability

Anne Brager, MS, RNProgram Supervisor

Frederick County Infants and Toddlers Program

Adapted from ECO Center

Good Outcomes Begin with Good Assessment!

What are Outcomes?• Outcomes are the benefits that children and

families experience as a result of early intervention

• The three child outcomes, as measured for accountability purposes are different than IFSP outcomes in that:– OSEP’s three child outcomes for accountability apply to

all children and reflect global functioning in broad areas of development (social-emotional, knowledge and skills, getting needs met)

– IFSP outcomes are specific to an individual child, based on his or her individual needs

What is Assessment?

“Early childhood assessment is a flexible, collaborative decision-making process in which teams of parents and professionals repeatedly revise their judgments and reach consensus about the changing developmental, educational, medical and mental health services needs of young children and their families.”

(Bagnato and Neisworth, 1991)Quoted in DEC Recommended

Practices, 2005

Assessment is…

“Assessment is a generic term that refers to the process of gathering information for decision-making.”

Mary McLean, 2004

Purposes of Assessment

In the early intervention systemassessment serves several purposes: • Identification of children with special needs

(including screening, diagnosis, and determination of eligibility for services)

• Planning instruction or intervention• Monitoring of the child’s level of knowledge or

progress• Program evaluation and accountability

Accountability???

• Accountability involves embedding ongoing data collection into program practices because the need to examine the effectiveness of the services being provided is ongoing

• Given the number of factors that influence the provision of services (e.g., staff changes, reimbursement formulas), it is not reasonable to assume that a specific program that was effective at one point in time will always be effective

Assessment for Accountability

• Policymakers and funders are raising legitimate questions about which programs are effective as they struggle with how to allocate resources

• The Government Performance and Results Act (GPRA) means Part C Programs need to provide evidence that program participants demonstrate positive outcomes

• An important issue for assessment is that the Office for Special Education Programs (OSEP) decided not to require states to use the same assessment tool. Each state is free to chose their own

Assessment as a Process

It is important to distinguish betweenassessment as a process andassessment as a single instrument or

test!• Assessment as a process involves multiple

ways of collecting information about a child’s performance, including, for example, observation, documentation of the child’s work, checklists, rating scales, portfolios and norm-referenced or curriculum based tests

Variation in What Tools are Used

So far these are the tools we know that

states are using for accountability:• Of 28 states who listed specific assessment

instruments:• HELP-15 states• BDI/BDI-2-13 states• AEPS-11 states• Creative Curriculum-6 states• ELAP-6 states

• Not yet determined-23 states

Variation in HowAssessments Have Been UsedIn the past, there has been much variationin assessment procedures in earlyintervention programs across the country:• In some programs, the only formal

assessment has been for eligibility• Other states completed assessment annually• Some states have always completed

assessment at exit

Views on Assessment

There have been major changes in last 15

years in how assessment of youngchildren is viewed by the early

childhood community:• Old position: Do not test little kids• New position: Ongoing assessment is part

of a high quality early childhood program

“The best way to understand the development of children is to observe their behavior in natural settings while they are interacting with familiar adults over prolonged periods of time”

--Uri Bronfenbrenner

New Thoughts on Assessment

What Changed our Thinking

The purpose of assessment was redefined

in the last decade:• It’s no longer about sorting, labeling, using to deny access to

services• It’s now about getting a rich picture of what children can do

and can’t do and using that information to help them acquire new skills

• It’s now about monitoring progress• New and different tools became available for general

early childhood: Curriculum-based assessments were developed, e.g., Creative Curriculum, Work Sampling, etc.

What Changed Our Thinking

• Assessment had always been seen as a process with multiple purposes

• Distinctions have been made between good and bad uses of assessment with young children

• Good assessment practices are now widely recognized as a key component of high-quality early childhood programs

• Good uses are now promoted!• For more information: National Association for

Education of Young Children (NAEYC) web site (Position statement on Curriculum, Assessment and Evaluation)

NAEYC Position Statement

Policymakers, early childhood professionals and otherstakeholders in young children’s lives have a sharedresponsibility to:• Make ethical, appropriate, valid and reliable assessment a

central part of all early childhood programs. • To assess young children’s strengths, progress,

and needs, use assessment methods that are • developmentally appropriate • culturally and linguistically responsive • tied to children’s daily activities • supported by professional development • inclusive of families

NAEYC Position Statement continued…

Connected to specific, beneficial purposes:1. Making sound decisions about teaching and

learning2. Identifying significant concerns that may

require focused intervention for individual children

3. Helping programs improve their educational and developmental interventions

DEC Recommended Practices

for AssessmentThe Recommended Practices emerged fromfocus groups and are supported by literaturefrom the Division of Early Childhood (DEC):• Parents and professionals who participated in these

focus groups identified repeatedly two concerns1. Principle stakeholders, parents and family

members must play a vital and indispensible role in assessment from beginning to end

2. Assessment methods and materials must accommodated children’s developmental and disability specific characteristics

DEC Recommended Practices

for AssessmentThe Recommended Practices are organizedaround five statements:1. Professionals and families collaborate in planning and

implementing assessment2. Assessment is individualized and appropriate for the

child and family3. Assessment provides useful information for intervention4. Professionals share information in respectful and useful

ways5. Professionals meet legal and procedural requirements

and Recommended Practice guidelines

Assessment for Accountability

The assessment process we use to determine eligibilityand program planning can be used for accountability!The three child outcomes:

– Children have positive social-emotional skills (including social relationships)

– Children acquire and use knowledge and skills (including early language/communication [and early literacy])

– Children use appropriate behaviors to meet their needs

• Assessment tools can inform us about children’s functioning in each of the three outcome areas

• Challenge: There is no assessment tool that assesses the three outcomes directly

Additional Challenges

• Young children are unreliable test takers • Because of their limited ability to understand

and follow the directions of the test• They may not be able to control their behavior• They have short attention spans• And are easily distracted

• What young children demonstrate in a test situation may not be an accurate representation of what they are capable of doing on a daily basis

Types of Tests

• Standardized vs curriculum based• By definition, standardized tests must be

administered the same way to all test takers.• The use of a consistent approach during test

administration enhances reliability between administrators and confidence that children are responding to the same tasks.

• Standardized tests usually provide normative data to compare the child’s functioning with that of other children of the same age.

Standardized Tools

Few standardized assessment toolsprovide detailed guidelines for

appropriateaccommodations during testing toeffectively measure capabilities amongthe broad range of children who havespecial needs.” McLean,

2005

Curriculum-based Assessments

• Bagnato and Neisworth (1991) define curriculum-based assessment as a “form of criterion-referenced assessment wherein curricular objectives act as the criteria for the identification of instructional targets and for the assessment of status and progress”

• Some of the newer assessment tools emphasize critical skills to facilitate the development of fuctional behavior

Currently Available Assessment Tools• Most assessment tools are organized around

developmental domains and are based on sequences of skills that build on each other

• This organization may pose difficulties for accurate assessments of functioning in young children with disabilities because delayed skills in one domain can influence assessment of or functioning in other domains

• Each assessment tool sees children through its own lens• Each lens is slightly different• Lens are not right or wrong

Variations Among States

• States are continuing to make key decisions related to their accountability systems, so any information about state choices must be seen as time-limited and interpreted cautiously

• In many state, some assessment-related decisions are being made by local programs, so there is variation within the states

• The information available thus far suggests that much of the data on child outcomes will be collected with curriculum-based tools

Variations Among States

• Most states are building their accountability systems around assessments that were already being administered for other purposes (such as eligibility determination or program planning)

• Few have adopted a new statewide measure solely for collecting data for accountability purposes

Maryland’s Top Five

• Following a survey distributed to all local Program Directors in the state, Maryland state Department of Education (MSDE) identified the 5 most commonly used assessment instruments in Maryland.

• Results from at least one of these assessment instruments (or other instrument if more appropriate), in conjunction with information obtained from other possible sources, including record review, interview, observation, and screening, will be used in providing the information required to complete the Child Outcomes Summary Form (COSF).

• Ideally, the same tool would be used at entry and exit. However, use the tool that is most appropriate for the child.

Maryland’s Top Five

• The five most common assessment measures are:– Early Learning Accomplishment

Profile (E-LAP)– Early Intervention Developmental

Profile (EIDP)– Hawaii Early Learning Profile (HELP)– Battelle Developmental Inventory II– Ages and Stages

Key Questions for Accountability:

1. How much and what information will the tool provide about the child’s current level of functioning and the attainment of the 3 child outcomes?

2. Would providers intentionally or unintentionally give children higher scores than warranted, knowing that conclusions about program effectiveness would be based on the information?

Assessment Tool Lens

• Each assessment tool carries its own organizing framework

• Most are organized around domains but children integrate skills when performing functional activities

• No right or wrong lens:– How can what the assessment tool provides be

used to develop functional outcomes for young children?

– How can what the assessment tool provides be used as information about the achievement of the three outcomes?

Critical Assumptions Related

to the 3 Child Outcomes• Achievement of the outcomes is age

based, children of different ages will demonstrate achievement in different ways

• There are many pathways to competence for children with atypical development (using sign language, wheelchair). This seems obvious but can get lost when an assessment tool uses a different lens.

Remember This

• Flexibility is required in using assessment tool results to make decisions about a child’s functioning on the outcomes

• Team needs to decide what information from an assessment tool is relevant to understanding child’s functioning

More DEC Recommended Practices for Assessment

Assessment provides useful informationfor intervention!• Assessment results are immediately useful for

planning program goals and objectives• Good assessment practices assess not only

immediate mastery of skill, but also whether child can demonstrate skill consistently across settings and with other people

• Good assessment practices also assess the presence and extent of atypical child behavior that may be a barrier to intervention and progress

Using the Assessment Results for Outcomes

What is an Outcome:• Outcomes are the benefits that children and

families experience as a result of early intervention• The three child outcomes, as measured for

accountability purposes are different than IFSP outcomes in that:– The three child outcomes reflect global

functioning in broad areas of development (social-emotional, knowledge and skills, getting needs met)

– IFSP outcomes are specific to an individual child, based on his or her individual needs

Let’s Look at Using Assessment Information to Develop Functional IFSP OutcomesWhat Makes an Outcome Functional?• Functional Outcomes:

– Refer to things that are meaningful to the child in the context of everyday living

– Refer to an integrated series of behaviors of skills that allows for the child to achieve the important everyday goals

Meaningful Outcomes

Meaningful outcomes promote a child’s functioning in three key foundations of early development - social interaction, mastery over environment, and engagement in learning - in ways specific to each family and child (McWilliam, 2002; National Research

Council, 2000).

Writing IFSP Family Centered, Measureable, Functional Outcomes

• Family-centered outcomes are written to address:• The desires and goals a family has for them

and their child• Created with the family after assessing what

activities are meaningful to individual family members.

• Includes not only activities of daily living but also the family's ability to participate in cultural and social experiences they hold valuable

Functional Outcomes

• “Function” refers to those activities identified by the family that support the development of the child’s physical, social and psychological well being, i.e. the ability to feed oneself, to hold a toy, to communicate, or to play

• Early intervention teams do not “treat” the primary diagnosis, i.e. cerebral palsy, autism, but instead treat the functional delay or disability

Determining Appropriate IFSP Functional Outcomes

A few questions to consider:• What are the activities the family would

like to do that are difficult?• Has the family given up on any family

outings, experiences or chores?• What kinds of things would the family like

for the child to do that would make life easier for them or more fun for the child?

Read more at MD’s IFSP Tutorial: Assisting Families to Select Functional Outcomes

Measureable IFSP Outcomes

• Outcomes should be written so that all members of the team will understand when the outcome has been met.

• Outcomes should address one of the following areas:• They should enhance the family’s ability to care for

or to engage in activity with their child• They should enhance the child’s ability to participate

in functional activities (feeding, dressing, moving in his environment, communicating, playing, etc.)

• They should expand on activity settings in which the child already participates

Read more at MD’s IFSP Tutorial:Measurable Criteria: How we Know we have Achieved Outcomes

Effective Strategies on an IFSP Identify How Functional Outcomes Will be AchievedEach strategy should address:

• Individualization- Do strategies build on child and family interests?

• Context- Do strategies build on familiar places, people, and routines?

• Mastery- Do strategies ensure generalization of a child’s actions and interactions in multiple settings and tasks?

• Collaboration- Do strategies specify who will do what?

Read more at MD’s IFSP Tutorial: Effective Strategies Identify How Functional Outcomes will be Achieved

Making a Difference!

The results of improved assessment andindividual IFSP outcomes developmentand services is two-fold:

– First and foremost, it ultimately leads to enhanced services for children and families

– Which, as a result, can’t help but positively impact Maryland’s outcome measurement results!

More to Come…

Next week: • Explore Maryland’s approach to

gathering child outcome data to measure OSEP’s Child Outcomes

• The role of the Child Outcomes Summary Form

• Explore other state’s approaches