Feb 22, 2016
Linking Early Intervention Quality Practices With Child and Family Outcomes
Technical Assistance for a Local Early Intervention SystemInfant & Toddler Connection of Virginia1Measuring Family OutcomesNCSEAM Survey administered annuallySurvey includes two rating scalesImpact on Families (22 items)Family Centered Services (25 items)Use of Part C standards (cut-scores)4a 5394b 5564c - 516
Infant & Toddler Connection of Virginia2We use the NCSEAM survey to measure family outcomes. This survey includes two rating scales, one of which measure impact on families and the other measures the quality of family-centered services provided to families. While both scales provide valuable information, only the impact on families scale is used for calculating the states results for the family outcomes. Deriving a percent from a continuous distribution requires application of a standard, or cut-score. Virginia uses the Part C standards recommended by a nationally representative stakeholder group convened by NCSEAM. The recommended standards, established based on item content expressed in the scale, were as follows: for Indicator 4a, know their rights, a measure of 539; for Indicator 4b, effectively communicate their childrens needs, a measure of 556; and for Indicator 4c, help their children develop and learn, a measure of 516.
The survey administered by the State of Virginia included two rating scales developed and validated by the National Center for Special Education Accountability Monitoring (NCSEAM). The 22-item Impact on Family Scale (IFS) measures the extent to which early intervention helped families achieve positive outcomes, including the three outcomes specified in Indicator #4. The 25-item Family-Centered Services Scale (FCSS) measures the quality of family-centered services provided to families.
For each scale, the analysis produces a measure for each survey respondent. Individual measures can range from 0 to 1,000. For the IFS, each familys measure reflects the extent to which the family perceives that early intervention has helped them achieve positive family outcomes. The IFS measures of all respondents were averaged to yield a mean measure reflecting the overall performance of the state in regard to the impact of early intervention on family outcomes.
Local Results: Percent Meeting StandardInfant & Toddler Connection of Virginia3This slide shows the significant slippage that occurred in one local system from 2009 to 2010. Because of this slippage, we provided targeted technical assistance to this system.Comparing 2009 LS Results to Targets and State ResultsInfant & Toddler Connection of Virginia4These next two slides show the relationship of the local results to the target and the state results. In 2009, the state results exceeded the targets, and the local system results were slightly below the target.Comparing 2010 LS Results to Targets and State ResultsInfant & Toddler Connection of Virginia5In 2010, the state results again exceeded the target, but the local system were dramatically below the targets. Local Results:Percent of TargetInfant & Toddler Connection of Virginia6This slide shows the slippage in a different way. You can see that though the results for indicator 4c were not dramatically different from 2009 to 2010, the local systems percent of the target is quite different for 4a and 4b for 2009 and 2010.TA Site Visit: Meeting OutcomesLocal staff will understand:relationship of the Survey to the Outcomeshow the survey results are converted to indicator resultsLocal Staff will develop hypotheses about slippageTools for analysis of results and improvement planning will be introducedNext steps will be determined
Infant & Toddler Connection of Virginia7A site visit was scheduled to provide targeted TA for the local system staff with an emphasis on inclusion of providers, especially service coordinators. The objectives for this TA visit included.TA AgendaCheck Your KnowledgeRelationship of Survey to Family Outcomes ResultsTranslating Survey ResultsComparison of 2009 and 2010 resultsAnalysis and improvement planning toolsNext stepsInfant & Toddler Connection of Virginia8Our plan for the session includedCheck Your KnowledgeWhat are the Family Outcomes?How are the results for the Family Outcomes Indicator derived for the Annual Performance Report to OSEP?What do the results measure? Who can impact the family outcomes?What factors impact the family survey results?
Infant & Toddler Connection of Virginia9A simple pre-test indicated that there was a big knowledge gap regarding family outcomes, including the relationship of the survey to the results and even a lack of information about the questions on the survey.Relationship of Survey to Outcome ResultsReview of the survey itself to understand what parents are being askedTwo scalesImpact on FamiliesFamily Centered practicesInfant & Toddler Connection of Virginia10So we started with a review of the survey including the information I mentioned earlier about the two scales. A number of the service coordinators and service providers did not know what families were being asked on the survey.
Survey ProcessRole of Local SystemProvide names/addressesInform families they will be receiving surveyProcess for disseminating surveyPre-survey notificationSurvey mailed to families
Infant & Toddler Connection of Virginia11We discussed the process for administration of the survey including the role of the local system and the process for dissemination.
Local system managers provide names and addresses of families whose children received services during the targeted time period. We contract with Old Dominion University to mail out the surveys and for analysis of the results.
Translating Survey Results to Outcome ResultsDetailed Analysis ReportAnalysis of the Virginia Family Survey Data Addressing Part C SPP/APR Indicator #4: Final Report 2010Summary Explanation 2010 Family Survey Explanation of Results
Infant & Toddler Connection of Virginia12ODU prepares a detailed analysis report about the results which is posted on our website. A summary explanation is provided by Part C staff and is also posted on the website.Reviewing /UnderstandingLocal ResultsItems at the bottom of each scale are more likely to be answered positivelyIdentification of areas needing improvementRelationship of practices to results
Infant & Toddler Connection of Virginia13The TA included an explanation of how the survey results are translated to family outcome results. For more details about this explanation, you may want to look at the Family Survey Talking Points handout. This handout was revised after the local system site visit and was used by the three TA Consultants to provide consistent information at regional meetings across the state by the three TA consultants. As part of this statewide TA, we provided a talking points document that the system managers could use to provide information and explanations to their local system service coordinators and service providers.Factors PotentiallyImpacting ResultsChange in Local Lead AgencyNew local system leadershipNew service coordinatorsNew direct service providersIncreased child count106 in 2009; 140 in 2010Statewide system transformationInfant & Toddler Connection of Virginia14During the March local system TA visit, we brainstormed issues that could be impacting the results. There were some very significant challenges for this local system including..Analysis and Improvement Planning ToolsLocal Contributing Factor Tool (RRCP/NECTAC/DAC (6-5-09) adapted for TA with local systemNECTAC/ECO Center: Relationship of Quality practices to Child and Family Outcome Measurement Results (draft 2/4/11)
Infant & Toddler Connection of Virginia15There were two major tools we used which are included in your handout. The first tool provided guiding questions to consider the impact of system/infrastructure factors as well as provider practices on family outcomes.
The 2nd tool, the focus of this session had just become available. It provided very concrete information not only about quality practices, but also about the relationship of specific practices to the child and family outcomes.Infant & Toddler Connection of Virginia16
The tool was reviewed briefly during the session with a request that each individual review it in more depth after the meeting.Next Steps: Local Follow Up PlanReview of ToolsFurther analysisStaff discussion and self-reflection Improvement Planning and ImplementationInfant & Toddler Connection of Virginia17The plan after the site visit included a continuation of the discussion, analysis and planning to be led by the local system manager and to include staff (SC and providers) during staff meetings.
Actions Taken by LocalityRevision of Intake FormStaff discussion/work on effective communication with familiesContinuing work to improve IFSP outcome developmentParticipation with child/family in activities/routines (store, swimming, etc)
Infant & Toddler Connection of Virginia18The local system manager reported back in August that they utilized parts of this form to re-think how we do/present things. In regards to Practice #2, they revised our intake form in an attempt to get a better picture of the family and what their concerns and needs are.They also focused on Practice #5. They have been focusing on IFSP process for the past 6 months, from whom should attend to how to write outcomes, to post IFSP follow up. The team approach to the IFSP has improved. Providers really do want to take an active role in the IFSP process, which is reassuring to the family because they feel comfortable with them because theyve done the assessment. They continue to work on communication and reflection skills,