-
Narratives
Executive Summary
OVERVIEW: United Way for Southeastern Michigan (UWSEM), an
existing grant making
institution, is the intermediary of the Bib to Backpack (BtoB)
SIF initiative. BtoB is geographically
based in the Metro Detroit region of Michigan (Wayne, Oakland
and Macomb counties) and is
focused on Youth Development (Technology, Early Childhood
Education) This tri-county area faces
many economic, social and educational challenges, with
particularly acute need in Detroit and
Pontiac. Subrecipients will include non-profit agencies that
have deep roots in tri-county communities
of need and understand the cultural, environmental and social
context our families are facing. Wayne
State University, Social Solutions, and Help Me Grow-Michigan
will each play a key role in
implementation of strategy. UWSEM requests $1.5 million for each
of 5 years. Sources of
commitment of match include Community Impact funds directed to
BtoB and UWSEM Campaign
funds.
PROJECT SUMMARY OVERVIEW: The proposed SIF, Bib to Backpack,
impacts society on three
levels. From a systems-building perspective, we will work with
subrecipient organizations to expand
programming, strengthen connections between programs, and
increase outcomes. From a
community and family perspective, we will support skill-building
for parents and other adults who
care for young children to empower those adults to expand their
social and resource networks and
increase their self-sufficiency and use of resources. All this
work converges on the third level, that of
young children, improving their developmental readiness and
setting them up for lifelong success.
THEORY OF CHANGE: We will build a parenting continuum that
recruits, engages, and
empowers parents to improve their family's self-sufficiency and
parenting skills and, in turn, their
children's developmental readiness with the following framework:
(1) Recruit eligible families through
a network of new and existing partners and systems into The
Incredible Years, REDI-P adapted Play
& Learns and Baby Box; (2) Co-create a plan with the family
to address concerns identified by
engaging the families with the Family Check-up model, the
Arizona Self-Sufficiency Matrix and the
Ages and Stages Questionnaire; (3) Use our existing 211
community-resource call center and 211 Care
Advocates to create a 'light-touch' care coordination system to
help families stay on track and
navigate the early childhood ecosystem; (4) Continue to monitor
participant families annually to
discuss progress, new priorities, and potential supports; (5)
Evaluate the impact of the integrated early
childhood (EC) system on families' self-sufficiency and
developmental readiness scores.
SUBRECIPIENT STRATEGY: A clear and effective Request for
Qualifications (RFQ) process will
include inviting agencies in Macomb, Oakland, and Wayne counties
to bid on work based on our
For Official Use Only
Page 2
-
Narratives
Theory of Change. Selection will depend on a scoring rubric and
an external review panel. Clear goals
and targets that support the end-goals and the evaluation of
same will be included in contracts and
reviewed. Continuing funding will be based on performance. UWSEM
has a proven track record of
effective subrecipient growth by taking an asset-driven
perspective and applying technical assistance
and capacity-building supports in order to expand upon an
agency's strengths.
UWSEM'S ASSETS: UWSEM has in place a highly-effective, robust
early childhood department,
including a network of Early Learning Communities that achieve
early childhood outcomes with early
learning resources, technical assistance, training, evaluation
support and funding. UWSEM's
experience, accomplishments and outcomes operating and
overseeing programs in Technology
include leading 211, Michigan's largest community resource Call
Center. We also have two Social
Solutions Efforts to Outcomes databases supporting our EC work
and have several staff capable of
adapting the software to new data and reporting needs. UWSEM's
internal Data and Evaluation Team
manages central data collection and evaluation activities for
our partners and for 211. Staff and
consultants are richly experienced and well-educated in a wide
range of disciplines.
Program Design
(b) PROGRAM DESIGN
(i) RATIONALE AND APPROACH
Intro: The United Way for Southeastern Michigan (UWSEM) believes
that every family has the
right to the supportive networks and resources that middle- and
upper-income families take for
granted. If we can increase access to supportive networks and
resources for families in at-risk
communities, then the families and children in those communities
will be more resilient. Parents and
other adult family members will be self-sufficient. Children
will be healthy, valued and thriving.
We take a multigenerational approach to the work of early
childhood (EC) well-being and
developmental readiness. By ensuring that the adults who care
for children have access to the
supports they need, and that they are empowered to make the best
use of those supports and to be the
voice for their young children, those adults can take better
care of their children and themselves.
The proposed SIF builds on our existing early child work, "Bib
to Backpack", and impacts society
on three levels. From a systems-building perspective, we will
work with subrecipient organizations to
expand programming, strengthen connections between programs, and
increase outcomes. From a
community and family perspective, we will support skill-building
for parents and other adults who
care for young children to empower those adults to expand their
networks and increase their self-
sufficiency and use of resources. This work all converges on the
third level, that of young children,
For Official Use Only
Page 3
-
Narratives
improving their developmental readiness and setting them on a
trajectory for lifelong success.
This body of work is conceived of as a continuum -- a series of
discrete elements that lead to a
transformational change (Overton, 2013) for families and
children. We will build out a continuum
that starts with pregnant mothers and ends with a connection to
school entry. Continuum approaches
are ways to track, analyze, and build connectivity among
families and organizations to influence
systems for successful results (Bailey & Terry, 2006). The
welcoming, intentional pathways of BtoB
will empower our target populations, pregnant mothers and the
adults who care for young children,
to easily access a series of evidence-based resources that will
lead to positive outcomes for the children,
the adults, and our communities.
STRATEGY B: The proposed BtoB SIF will be a geographically based
program focused on Youth
Development throughout Michigan's Wayne, Oakland, and Macomb
counties. Within this region, it is
our mission to ensure that children 0-5 are healthy, valued, and
thriving and we believe this happens
as a result of knowledgeable, responsive, and integrated
parenting. Our goal, specific to that mission,
is to build a parenting continuum that will recruit, engage, and
empower parents to improve their
family's self-sufficiency and, in turn, their children's
developmental readiness. By building this system
on a foundation of evidence based parenting strategies and
technologically driven care-coordination
supports we believe we can both educate and empower parents to
improve the developmental
readiness and well-being of their children.
Theory of change: Build a parenting continuum that will recruit,
engage, and empower parents to
improve their family's self-sufficiency and parenting skills
and, in turn, their children's developmental
readiness. This will be accomplished by using the following
framework: (1) Recruit target-population
participants through a network of new and existing partners and
systems into The Incredible Years,
the Research Based Developmentally Informed Parenting Program
(REDI-P) adapted Play & Learns
(REDI-PPL) and Baby Box (BB); (2) Once they are enrolled in
programming, engage interested,
eligible families with the Family Check-up model (FCM) and
perform full family self-sufficiency and
developmental screenings using the Arizona Self-Sufficiency
Matrix (ASSM) and Ages and Stages
Questionnaire (ASQ) and co-create a plan with the family to
address immediate concerns; (3) Use a
'light-touch' care coordination system to help BtoB SIF families
stay on track and navigate the early
childhood ecosystem. We will use our existing 211 system and 211
Care Advocates to help monitor
participant progress and make additional referrals based on the
families' plans; (4) Continue to
monitor participant families annually to discuss progress, new
priorities, and potential supports; (5)
Evaluate the impact of the integrated EC system on families'
self-sufficiency and developmental
For Official Use Only
Page 4
-
Narratives
readiness scores.
CRITICAL PROBLEM (RATIONALE): The tri-county area in
Southeastern Michigan faces many
economic, social and educational challenges: almost half (48%)
of the estimated 236,511 children 0-4
years of age live below 200% of the federal poverty rate.
Detroit's unemployment rate at 11% is twice
the national average. County unemployment rates are more in
line, as the Bureau for Labor Statistics
reports them in March 2016 at 6.4, 4.3 and 5.7 percent in Wayne,
Oakland and Macomb, respectively.
While economic challenges are prevalent throughout the region,
the need is particularly acute in
some communities, including the cities of Detroit and Pontiac.
Detroit has a poverty rate of 40%, with
two-thirds of children under 18 living at less than 125% of
poverty (U.S. Census Bureau, Table S1703,
2010-2014). Of adults over the age of 25, 22% do not have a high
school diploma (compared to 13.6%
nationally; U.S. Census Bureau, Table S1501, 2010-2014). In
Pontiac, conditions are similar with a
38% poverty rate and 61% of children under 18 living at less
than 125% of the federal poverty rate
(U.S. Census Bureau, Table S1703, 2010-2014). Of adults in
Pontiac, 23% do not have a high school
diploma (U.S. Census Bureau, Table S1501, 2010-2014).
Language will also be an important consideration for
implementing the proposed SIF initiative. In
Macomb County, 13.5% of the population speaks a language other
than English, with 42.8% speaking
English "less than very well". In Oakland and Wayne Counties,
14% and 37.2% of the population
speaks a language other than English respectively, with 32.6%
and 37.2% speaking English "less than
very well". (U.S. Census Bureau, Table S1601, 2010-2014).
The social and economic challenges faced by our region have many
implications for the health and
developmental readiness of our young children. Infant mortality
remains a great community concern
as Detroit is over twice the national average with 13.2 infant
deaths per 1,000 reported, compared to
5.9 nationally. There are 10.3 infant deaths per 1,000 reported
in Pontiac and the infant mortality
rates in the three counties, Wayne, Oakland, and Macomb are 9.6,
6.3 and 5.8 respectively (Michigan
Department of Health & Human Services, 2010-2014).
The developmental readiness of our region also remains an area
of deep concern. UWSEM heavily
invests in and promotes the use of the Ages and Stages
Questionnaire (ASQ), a screening tool that
monitors young children's (ages 0-5) development and focuses on
children's general and social-
emotional development. Investment in these screenings is
important because children living in high-
risk communities suffer the most frequent and the most severe
developmental delays and take longer
to recover from delays than children living in other
communities. High-risk communities are
generally characterized by concomitantly high rates of poverty,
unemployment and violence.
For Official Use Only
Page 5
-
Narratives
71,749 Ages and Stages Questionnaire screens have been completed
on children 0-5 as of
September 2015 in the three counties and confirm this finding.
In each domain, over a third of
children screened were at risk. The following is a breakdown of
percentage of children at strong risk or
potential risk, combined, of each developmental domain:
Communication- 36%, Gross Motor- 38%,
Fine Motor- 54%, Problem Solving- 38% and Personal/Social- 35%.
According to James Heckman,
Nobel Laureate in economics, addressing these deficiencies early
has vast returns both socially and
financially for both the children and society
(www.heckmanequation.org).
CURRENT SOLUTIONS NOT WORKING: In these at-risk communities,
there is no system that
readily connects people to all of the resources they need to
support them as parents and to meet their
objectives as families. While many resources are available, they
are sometimes underutilized and there
is a lack of continuity and connectedness to ensure that
families can easily identify and access the
specific supports they need. Parent support programs exist in
Greater Detroit, yet they are not
coordinated and connected in a way that can be monitored to
better understand how to best serve our
community and achieve outcomes at scale, while efficiently
stewarding our resources.
Although we have deep regional penetration around ASQ
screenings, following-up and connecting
parents and caregivers with referrals and ongoing support
remains a challenge and an area that
requires further investment. While each county has active pilots
to better understand how to track and
support connections to referred services, few have the staff
capacity to make sure that the referrals
lead to connections. Creating a connection to our 211 system
under this pilot would fund our Care
Advocates to track our participants using Help Me Grow (HMG; an
ASQ implementation
organization) services and perform the follow-ups necessary to
ensure connection to services, while
allowing HMG staff to focus on building their network of
referral agencies and analyzing screens for
referrals.
We see a similar issue in our own 211 call center. Our call
center is a 24/7 referral service that
connects people with information and resources to build healthy,
safe communities. The center serves
eight (8) counties in southeastern Michigan, including Wayne,
Oakland and Macomb counties. Every
year the center receives approximately 300,000 calls from
individuals needing assistance. 211
maintains a validated database of approximately 12,000 programs
and 2,000 organizations and
agencies.
There were a total of 10,347 calls from individuals seeking
individual, family, and community
assistance, not including. However, not all of these calls
result in a referral for the individual for a
variety reasons. 7.7%, 18.8%, and 9.5% of calls seeking
individual, family, and community assistance
For Official Use Only
Page 6
www.heckmanequation.org).�
-
Narratives
calls ended with no referral being made for Wayne, Macomb and
Oakland respectively.
Additionally, not all individuals referred by 211 receive
services from the agencies they were
referred to. On a 2015 211 follow-up survey comprised of 1,125
respondents, 69.4% reported not
receiving services from the referred agencies. The most common
reasons respondents reported not
receiving services were 1) no funds available (17.8%), 2) agency
determination pending (13.8%) and 3)
Full/At Capacity/ Waitlist (11.4%). By maintaining consistent
contact with B3 families we hope to
keep them engaged with our system to work through these barriers
in partnership.
Each of these programs are largely successful at what they were
designed to do and we know that
with additional coordination supports we can bolster their
effectiveness with our most at-risk families
through the approach outlined below.
APPROACH - PROPOSED EVIDENCE-BASED INTERVENTION COMPONENTS:
These are
descriptions of the core, evidenced-based components of our BtoB
continuum. Please see the
evaluation section for the justification regarding their
evidence status; each core component is
currently either at a preliminary or moderate level of evidence.
The section following this describes
how the components are integrated and how a family will navigate
through the system under our
Theory of Change.
Family Check-Up Model (FCM): the FCM is a care management model
that involves a lighter
version of care coordination than many existing models. It
differs from traditional clinical models and
practice in three important ways: (1) it utilizes a health
maintenance model, (2) it derives much of its
power from a comprehensive assessment, and (3) it emphasizes
motivating change. In contrast to the
standard clinical model, the health maintenance approach of the
FCM explicitly promotes periodic
contact with families (yearly at a minimum) over the course of
key developmental transitions (Gill, A.
et al, 2008), potentially reducing coordination costs.
The FCM facilitates change through two main components:
motivational interviewing and family
management practices. In working with families of young
children, the FCM motivational
interviewing session is designed to elicit motivation for the
parent(s) to change problematic behavior
in their child, which is often achieved by modifying parenting
behavior (Forgatch et al., 2005) or
aspects of the caregiving context that compromise parenting
quality. The motivational interviewing
focuses on sharing the assessment data results and working with
the family to identify potential
problem areas they wish to address (Gille, A., et al, 2009). Our
model includes use of the Ages and
Stages Questionnaire (ASQ), when appropriate, to empower parents
with additional developmental
readiness information and referrals aimed at improving or
maintaining their children's developmental
For Official Use Only
Page 7
-
Narratives
readiness. By using these techniques and focusing on priority
areas for the family we hope to increase
and maintain engagement in the SIF parent continuum.
After understanding a family's motivation for change, the FCM
provides options for intervention.
Options may include referrals for help with problems outside of
parenting, but the focus of most
interventions address family management issues. Family
management includes a collective set of
parenting skills, commonly referred to as Parent Management
Training (PMT), based on social
learning principles of reinforcement and modeling (Forgatch et
al., 2005; Patterson, 1982; Webster-
Stratton & Hammond, 1997). PMT has been consistently
associated with improvement in parenting
and reductions in child conduct problems (Bullock &
Forgatch, 2005; Patterson, Reid, & Dishion,
1992).
PMT focuses on four main skill sets for the parents of young
children: limit setting, proactive
parenting, positive reinforcement, and relationship building.
Using PMT typically involves providing
parents with a rationale to stimulate interest, careful
explanation of new skills, and in-session practice
using role plays and practice with the child. In the FCM, PMT is
applied to specific behavior problems
highlighted in the assessment. Our PMT interventions will
consist of the Incredible Years Training
(IYT) and Play and Learns adapted from the REDI-P home visiting
model (REDI-PPL), as well as
coordinated referrals into other UWSEM parenting classes and/or
community supports to deal with
specific issues that may not be covered in the IYT or
REDI-PPLs.
Incredible Years Training: This training is based on principles
of video modeling, observational and
experiential learning, rehearsal and practice, individual goal
setting, self-management, self-reflection
and cognitive self-control. It looks to reduce risk factors and
increase protective factors through
increasing parent, teacher, and child competencies. Most of our
recruited families will have self-
selected into the parenting classes and our focus will remain on
the parenting components of the IYT.
Numerous randomized control group trials have been conducted
over three decades by the developer
and independent investigators with parents in high risk
populations (e.g., Head Start and Sure Start)
showing positive outcomes for parents.
REDI-P - REDI-P is a parent-child interaction intervention that
is adapted from a preschool based
intervention. It involves a 16 "lesson" home visiting curricula
which will be adapted for this proposal
to be used in play-and-learn groups (REDI-PPL). This
intervention is considered to have preliminary
to moderate evidence as it has been evaluated with at least one
randomized control trial comprised of
200 ethnically diverse children and families recruited from
three counties in Pennsylvania. More
importantly, the components of the intervention are based upon
an extensive review of the literature
For Official Use Only
Page 8
-
Narratives
on positive parent-child interactions and their link to positive
child outcomes (Bierman et al, 2015).
Baby Box: Finland's universally-distributed Baby Box is
literally a box outfitted with a mattress
that can be used safely as a bassinet. It is credited with
helping to reduce that country's infant
mortality rate from 65 to 3 deaths per 1,000 children from 1938
to 2013. When distributed the Box is
filled with infant products and information linking new parents
to critical resources.
Baby Box University: A vital component of the Baby Box program
is Baby Box University. Expectant
and new parents have access to thousands of short, online
videos, in many languages, from
professionals talking about safe sleep, breastfeeding, positive
parenting and other important aspects of
caring for newborns. These educational opportunities are
available on mobile devices 24/7 and
engagement can be tracked. Research has shown that one of the
most universally impactful tools for
reducing child mortality is parent education.
THEORY OF CHANGE (TOC) and PROGRAM DESCRIPTION: Building a
parenting continuum
that will recruit, engage, and empower parents to improve their
family's self-sufficiency and parenting
skills will improve their children's developmental readiness. By
adopting Strategy B and the above
evidenced-based interventions we can exercise greater oversight
over the evaluation, increase the
chance of successful scaling by using well-known,
evidenced-based national models, and family
resiliency.
Initially, families will be recruited into the SIF intervention
by our partner agencies and
subrecipients and placed into one of two integrated pathways:
expectant mothers and existing families
(families with children 0-5 already in the home). Specifically,
we will recruit pregnant women,
parents, and families of children age 5 and under living at or
below 200% or less of the FPL who reside
in Wayne, Oakland, or Macomb County. Initial recruitment will be
based on a need recognized by our
partner organizations and the mother or family's interest in the
Baby Box and/or our two parenting
supports: Incredible Years Training and REDI-P Play and Learns.
All families participating in these
interventions will be prescreened for BtoB eligibility to ensure
they are within our target population.
Subrecipients are ideally either parent-centric or multifaceted
family services agencies with deep
roots in tri-county communities of need that understand the
cultural, environmental, and social
context that our target families are facing. As an existing SIF
intermediary, we have extensive
experience in identifying prospective partners whose work aligns
with our TOC, and providing value-
added activities and supports to support their success. We will
draw on this experience to ensure that
selected subrecipients are provided with capacity building
technical assistance including but not
limited to: finance and accounting systems, procurement
policies, National Service Criminal History
For Official Use Only
Page 9
-
Narratives
Check procedures, program goal setting and implementation, data
collection and reporting, and rapid
cycle evaluation support.
Existing families meeting the BtoB SIF eligibility requirements
will be recruited to fully participate
in the BtoB SIF program. Each BtoB SIF family will be engaged
with the Family Check-up Model
(FCM), including a full family assessment with portions of the
Arizona Self-sufficiency Matrix
(ASSM), a nationally vetted family self-sufficiency assessment,
looking at relative poverty and a
supplemental child developmental screening using the ASQ. These
assessments and screens will
inform the Family Plan that will be created in conjunction with
a trained FCM Specialist (FCMS)
using motivational interviewing to help parents identify their
family's priority areas as they discuss the
assessment results together.
At the end of the co-planning session, the FCMS discusses a menu
of family-based interventions
with the caregivers. These options may include (a) regular
follow-up support, either in-person or by
phone; (b) assistance with specific child behavior problems or
parent issues; (c) Parent Management
Training; (d) preschool/day care consultations; and (e)
community referrals with light care
coordination provided by specially trained 211 Care Advocates.
The FCMS encourages the parents to
choose the level and type of services that best meet the
family's needs.
Having completed the initial stages of the FCM, a family will
have identified focus areas that they
wish to work on during the next 6-12 months and UWSEM will
support their continued connection to
both evidenced-based SIF PMT models as well as other community
referrals. The families FCM/ASSM
and ASQ results and family plan will be recorded and their
activities will be tracked using the 211
system REFER and 211 care advocates. 211 care advocates, in
partnership with the FCMS, will
monitor a family's progress through a centralized system put in
place to send reminders, track PMT
activities, community referrals and connections, and other
messages to keep families engaged with the
system. A variety of strategies will be tested to understand and
improve the effectiveness of this
approach.
Text messaging was used successfully by nurse discharge planners
to remain in contact with new
mothers who were discharged early. (Boe et al., 2015). The study
suggested nurses were able to offer
support and information to mothers and their families in a way
that enhanced the families' sense of
security and self-efficacy. The intervention included text chats
with nurses as needed and automated
messages about their babies' first 4 days, delivered every 12
hours. The nurses regarded the automated
messages as a tool to stimulate both the families' curiosity and
capacity to take control of their
situation. The nurses believed that the interactive links in the
automated messages encouraged parents
For Official Use Only
Page 10
-
Narratives
to read additional information material in the knowledgebase or
question the nurses on the postnatal
ward directly through their phones. They also felt it helped
manage the vast amounts of information
provided at discharge (Boe et al., 2015). Both our 211 and ETO
databases can be configured for text-
based messaging and information delivery and we will work with
our IT department to design content
and messages around our BtoB focus areas, e.g. have you read to
your child today?, follow this link
for a free interactive story.
Additionally, research performed by Dr. Vishal Mehta relying on
randomized, controlled trials and
systematic reviews of the literature concludes that "there is
now convincing data that text messaging
can influence health behavior and decisions." UWSEM's Healthy
Kids staff have developed a similar
text-messaging system to help people find locations and set
reminders for meal times at their food sites
around MI.
Engaging Expectant Mothers: Expectant mothers will largely be
recruited using the UWSEM Baby
Box that will be promoted through partner organizations,
including hospitals, family service
organizations, WIC offices, Nurse Family Partnerships, Maternal
Infant Health programs, Early Head
Start programs and others who will inform expectant or new
mothers of the box and provide them
with a link to three mandatory Baby Box University classes
(approx. 5 minutes each). We will select
three pilot cities within our tri-county regions for universal
deployment, meaning there will be no
restrictions on who may receive a Baby Box. Universal deployment
is important because it removes
any implied stigma around receiving the box if every mother in
the three initial test cities receives one.
Upon completion of these classes, families will receive an
electronic certificate of completion which
they will present to a Baby Box distribution partner (all BtoB
partners will be Baby Box distribution
partners) and receive their box. BtoB sites embedded in our most
at-risk neighborhoods will then use
this contact point to recruit the mothers into BtoB SIF. Those
interested mothers screened as eligible
will be enrolled into the BtoB SIF program and the FCM for
additional assessment. It is expected that
UWSEM will distribute approximately 12,000 Baby Boxes each year,
casting a wide net for
recruitment.
In addition to the initial entry tracks of expectant mothers and
existing families, we have identified
three focus pathways that will be priorities as part of our
pilot continuum. These are connections to
health care services, including insurance and primary care;
connections to nutrition programming;
and connections to Help Me Grow's referral network. By
addressing each of these early we can allow
the family to begin to focus on other social determinants and
our parenting supports.
Connections to Primary Care: UWSEM recognizes that connection to
consistent and quality
For Official Use Only
Page 11
-
Narratives
primary care is important for both maternal and child health and
all BtoB SIF participants will be
screened on insurance and primary care status to determine if a
referral needs to be made. We are
actively engaged in a pilot test using 211 Care Advocates to
screen 211 callers for potential connection
to a primary care physician, health insurance, and other health
services through our Children's
Healthcare Access Programs (CHAP) partners. CHAPs provide
families with resources, programs,
services and specific health supports based on the callers'
family needs and interests. Under this model,
a 211 Care Advocate addresses the caller's immediate reason for
calling by making the needed referral;
however, the 211 CA will also ask probing questions to ascertain
if a CHAP connection would be
worthwhile. The 211 Care Advocate then gives the caller general
information about CHAP. Consent is
provided by the caller to get connected to a Virtual CHAP
Specialist or local CHAP program, based
upon the county they reside in. The CHAP partner then helps
connect the family to needed health
services. Under the BtoB model the connections and activities
supported by CHAP will be recorded and
monitored to ensure that referral connections are being made and
any barriers to services addressed.
Connections to Help Me Grow: We have a longstanding partnership
with Help Me Grow -- a
national affiliate organization supporting the use of the ASQ --
to provide screening and referral
services for families using the ASQ. They currently have a
network of approximately 400
organizations to help provide ASQ screenings and developmental
services for families. Aside from a
few pilots to track referrals to pre-school and Head Start,
their ability to monitor connections to
referred services is limited. Under this BtoB model if a family
chooses to connect with HMG as a result
of a ASQ screen, our 211 Care Advocate would be able to record
that referral and follow up to see if the
connection was made and work to remove any barriers if it was
not, sharing back this information
with HMG regularly to help maintain an active network of
agencies.
Connections to Nutrition Programming: Lastly, we intend to build
on the work of our very
successful Meet Up and Eat Up summer meals program. Research
indicates that proper parental
modeling during the toddler years plays a significant role in
establishing longer-term eating behaviors
(Birch et al, 2007). We intend to use Cooking Matters to help
our families learn how prepare healthy
meals on a budget. The first-ever long term study of Cooking
Matters showed the six-week cooking,
shopping and nutrition course has a powerful, sustained impact
that is significantly greater than
changes that would have occurred without an intervention. After
the course, families felt more
confident that they will be able to afford enough food, were
eating healthier, putting them at lower
risk for diet-related diseases like obesity, diabetes, heart
disease and cancer, and cooking meals more
often, and making meals healthier and more budget-friendly.
Additionally, there was a 10% increase
For Official Use Only
Page 12
-
Narratives
in families that are more confident in their cooking abilities
and an 11% decrease in barriers seen by
families to making healthy, affordable meals. Cooking Matters is
the nutrition education program of
Share Our Strength's (a current SIF Intermediary) No Kid Hungry
campaign, an initiative to end
childhood hunger in America.
ALIGNMENT TO CURRENT SIF: We are not applying for renewed
funding of our current 2011
SIF project, but UWSEM is a current recipient of other federal
funding, and this proposal is an
opportunity to supplement and integrate those existing bodies of
work into our continuum.
Specifically: (1) As a SIF intermediary (Youth Development), we
fund five subrecipients to test
innovative approaches to kindergarten readiness. Approaches
include family/child-centric
interventions, interventions focused on immigrant communities,
and a healthy eating intervention.
This 2016 SIF proposal is an opportunity to create a true
continuum that has the potential to connect
with the 2011 interventions. (2) As a SIF subrecipient of the
intermediary Share Our Strength
(Healthy Futures), UWSEM is establishing regional collaborative
tables to engage key local leaders in
each of the project's six geographic program target areas. They
are working to provide comprehensive
outreach for their "Meet Up and Eat Up" summer meals program,
contracting with breakfast coaches
to encourage alternative breakfast models, and recruiting
year-round sites to serve both summer and
after school meals. Eventual integration with this network is a
mid-term goal for our scaling efforts.
(3) As the state of Michigan's implementation partner for the
Race to the Top - Early Learning
Challenge in Macomb, Oakland, and Wayne Counties, we work
closely with licensed and unlicensed
childcare providers to ensure that children receive quality
care. This SIF proposal further supplements
that work by ensuring that parents are provided with the
supports they need to parent effectively and
understand the need for quality childcare.
Creating a coordinated system that can eventually incorporate
all of our UWSEM programming
will help support and expand each of these programs, by helping
ensure families that need the
supports offered by these programs connect with these
programs.
In creating a continuum of adult-focused supports for children's
earliest years, BtoB SIF will
ultimately incorporate and/or connect to all of the UWSEM's
interventions that make up our existing
body of work in early childhood, Bib to Backpack. With
appropriate data collection, care coordination,
and referrals, we will ensure that families empowered by BtoB
SIF are invited to other interventions.
Each intervention will become part of a fully-integrated system
supporting young children and their
families. Parents and other adult family members will be
self-sufficient. Children will be healthy,
valued and thriving.
For Official Use Only
Page 13
-
Narratives
ii. Proposal for Subrecipient Selection
SUBRECIPIENT PROFILE: Subrecipients best suited to implement our
approach would share a
profile that demonstrates the ability to implement
parent-focused interventions with fidelity.
Investments will be made in tri-county nonprofit agencies that
have the capacity to recruit and retain
participants, to undertake data collection, to help users
navigate through the social services system,
and to integrate referrals across the continuum for the best
possible user experience. Consistent with
our theory of change agencies will meet parents at whichever
door they enter, and engage and
empower them to best prepare their children to succeed.
PLAN FOR SUBRECIPIENT SELECTION: Our plan to carry out a
competitive selection process is
clear and comprehensive and is formulated based on extensive
experience in doing so successfully.
Subrecipient agencies are ideally either parent-focused or
multifaceted family services agencies with
deep roots in tri-county communities of need. We expect to fund
four to six agencies with grants
ranging from $150,000 to $300,000. This will allow for a total
programmatic investment of
$300,000 to $600,000 per subrecipient. Subrecipients will meet
the criteria outlined in Appendix 1.
Based on our eight years' experience implementing Early Learning
Communities in greater Detroit,
we are confident that budgets in this range will allow for
appropriate programming, data collection,
and referral support.
As an existing SIF intermediary, we have extensive experience in
identifying prospective
subrecipients whose work aligns with our Theory of Change, and
providing value-added activities and
supports to support their success.
Our plan for carrying out a clear and effective bidding process
will mirror our recent experience in
re-bidding our Early Learning Communities (ELCs) in 2015, using
a process similar to STRATEGY B.
We provided a Theory of Change and a complete Request for
Qualifications (RFQ) document to
agencies in Macomb, Oakland, and Wayne counties, inviting them
to bid to undertake work as an
evolved ELC. We modeled the entire process on SIF, including a
scoring rubric and an external
review panel.
A total of 13 agencies submitted bids. The review process was
thorough and included scoring each
bid against a rubric, writing a narrative summary of the bid and
the agency's relevant body of work,
and the convening of a volunteer Review Panel that reviewed all
these materials and provided
recommendations. A UWSEM staff member with the appropriate
background and expertise to enable
him/her to evaluate the bid content scored each rubric section.
The scoring was consistent in that the
For Official Use Only
Page 14
-
Narratives
same person scored the same section(s) of each bid. The Review
Panel included a member of
UWSEM's board, a program officer from a major funder of our
early childhood work, an external
expert on our region's early childhood system, and a UWSEM Team
member with expertise in
financial stability.
The process from release of the RFQ to final determination of
agency partners took just over three
months. Based on this experience, we are confident in proposing
the following timeline, so that the
selection process for this 2016 SIF will be complete six months
following the start of our SIF
intermediary award.
Timeline: September 2016: Intermediary award start date. Draft
Request for Qualifications
(RFQ), including Theory of Change, detail on specific
interventions to be implemented, target zip
codes, expected capabilities of subrecipient agencies, and
scoring rubric. October 2016: Solicit
volunteers for review panel, include review of potential
conflicts of interest; panelists are likely to
include representatives from area universities, hospitals,
foundations and parents. Release RFQ to
agencies operating in greater Detroit, including not just
current and past UWSEM-funded agencies
but all those that may be eligible to apply. November 2016:
Letter of Interest deadline with 25-30
LOIs expected. Technical assistance (TA) workshops (in-person)
conducted. December 2016:
Subrecipient bids due; 15-20 bids expected. Initial intake and
review of bids to ensure completeness
and eligibility. January 2017: Site visits by staff and review
panel. Convene review panel for final
funding recommendations. February 2017: Staff review of funding
recommendations, including
consideration of potential conflicts of interest. Notify bidders
and begin contracting process.
IMPLEMENT PROCESS: In addition to our current SIF initiative, we
have demonstrated a
selection process with a high likelihood of identifying high
performing subrecipients that meet the
criteria outlined in Appendix 1 through the process used to
select grantees for the Evolved ELC project.
The process included details on number of awards, funding levels
and associated criteria, range of
implementation, and evaluation stipulations. Bidders, in turn,
provided data-informed proposals
demonstrating a capacity to implement the RFQ with fidelity. TA,
written and disseminated, on-line
and in-person regarding the process was provided.
UWSEM has extensive experience selecting and awarding
competitive grants to non-profits. In the
last year, we awarded $12.2 million Community Impact investment
dollars in the tri-county area.
HIGH PERFORMING RECIPIENTS UNDER STRATEGY B: Under Strategy B,
we will select
subrecipients who are well suited to implement the predefined
intervention. Successful applicants will
demonstrate that the proposed interventions will be an integral
component of their agency offerings,
For Official Use Only
Page 15
-
Narratives
including demonstrating how the goals of the BtoB SIF are but
one part of a larger system in which
we all operate. Agencies will also be selected based on their
expected ability to implement the evidence-
based interventions with fidelity.
Commitment to families with young children will be demonstrated
by an agency's near perfect
alignment with UWSEM's three Early Childhood Outcomes. These
outcomes are children are
developmentally ready to learn; parents and caregivers nurture
children's development and
communities promote children's well-being. Agencies will
demonstrate that the Early Childhood
Outcomes are integral to their body of work as a whole and that
they raise awareness and inspire
action on interventions to ensure children age 0-5 are
developmentally ready to learn.
To demonstrate a commitment to BtoB SIF, an agency will
demonstrate a superior understanding
and alignment of BtoB with the agency's own mission. It will
demonstrate an extended and successful
history, knowledge, and experience serving pregnant women and
parents of young children. It will
show how this initiative directly supports or is related to the
organization's mission or vision. And it
will give clear example(s) of foreseen obstacle(s) and
solution(s) and how unforeseen obstacles have
been dealt with in the past.
A successful candidate will demonstrate a community presence by
showing a strong track record of
success in recruiting and retaining program participants.
Moreover, it will demonstrate a superior
understanding and alignment of recruitment and retention
strategy with other agency efforts.
The agency must be known for, and demonstrate, making newcomers
feel welcome and
encouraging parents and caregivers to return for additional
enrichment. It is identifiable and easily
recognizable as a resource to the community. The agency knows
the target population and provides
resources and content to meet their needs and creatively works
to ensure services are accessible.
Agency's facilities are known for being easily accessible and it
has measures in place to assess
participant safety and comfort while on site.
A subrecipient demonstrates the use of partnerships to deliver
deeper results or to reach more
participants. Providing a list of key partners will be required.
The agency helps to foster a cultural
change and movement around the importance of early childhood
interventions and the importance of
parent engagement and empowerment in the community.
Implementation fidelity and data competence are critical
attributes of the successful bidder. The
agency demonstrates a successful history of delivering program
components in ways known to
enhance effectiveness. It demonstrates a strong capability for
implementation fidelity for the model
component(s) the agency proposes to undertake. There is a
process in place for monitoring model
For Official Use Only
Page 16
-
Narratives
implementation and continuously checking program quality (e.g.
staff training on the model,
constant monitoring of staff and programs, etc.). The agency
demonstrates strong experience
monitoring program performance. Its strategy includes measures
for quality and impact and captures
participant feedback. Agency has accountability measures in
place for data quality and timeliness and
has strong experience with different methods of collecting data.
The subrecipient demonstrates
significant capacity for data collection and performance
management including necessary staff to
manage data collection, technology to manage and store data, and
policies/procedures to collect data.
Finally, the agency successfully disseminates knowledge and data
to other organizations and in a
manner relevant to the EC Network.
READINESS ASSESSMENT: Specific criteria for assessing applicant
readiness will include
considerations relevant to staffing, use of data, community
integration and more. As an existing SIF
intermediary, we have extensive experience in providing
value-added activities and supports to
subrecipient agencies. We would draw on this experience to
ensure that selected subrecipients are
provided with capacity building technical assistance including
but not limited to: Finance and
accounting systems, procurement policies, National Service
Criminal History Check procedures,
program implementation, data collection, and reporting.
iii. Proposal for Evaluation
CAPACITY TO ENSURE EVALUATION: Our capacity to successfully
evaluate our portfolio has
been demonstrated repeatedly. Specifically, we have extensive
experience in managing and supporting
evaluations of past funded program models, including our current
SIF program. This SIF experience
also demonstrates how UWSEM has supported recipients to improve
program performance, to apply
evaluation results to decision making and to successfully
complete evaluations of their respective
program models.
We are currently in year five of a current SIF grant in which
subrecipients are conducting rigorous
evaluations of programs designed to increase skills and
attributes related to school readiness among
children aged 0-5; increase the capacity for parents of at-risk
children aged 0-5 to support school
readiness in the home; and increase community level awareness of
the need to support early
childhood learning as well as increase formal and informal
processes through which communities
support families and children in achieving improved school
readiness.
The initial portfolio was comprised of 11 different sub-grantee
partners implementing a wide array
of intervention strategies. The sub-grantee selection process
was designed to ensure that interventions
For Official Use Only
Page 17
-
Narratives
ranged from community based primary prevention efforts, to
secondary prevention interventions
focused on early childhood initiatives implemented through
preschool, head-start, and parenting
groups, to secondary and tertiary prevention efforts focusing on
intervening directly with families in
the home.
UWSEM developed a technical support system and review process,
which led to the successful
development of evaluation protocols that were able to meet at
least moderate levels of evidence by
year 4 of the SIF programming for all subrecipients. These
evaluations involve rigorous evaluation
designs including Randomized Comparison Groups at both
individual and group levels, Waitlist
Randomized Controls, and Propensity Score Matched Control
Designs. In addition, all but two of the
evaluations incorporate multiple post-test repeated measures
designs in conjunction with
experimental or quasi-experimental control groups.
UWSEM currently has a portfolio of 5 SIF sub-grantees which have
successfully implemented
sophisticated and rigorous evaluations across a 4 year period.
In addition, an overall portfolio
evaluation has successfully integrated the individual SIF
evaluation data into a comprehensive
overview of SIF impact across the entire portfolio.
UWSEM has implemented a number of strategies for managing and
supporting evaluation efforts
at both the portfolio and sub-grantee level. These include:
Monthly Evaluator Network meetings,
bimonthly Learning Community meetings integrating SIF program
discussions with evaluation
discussions, an open system of continuously available technical
support for all aspects of evaluation,
annual SIF summits through which, best practices, lessons
learned, and resources are shared across
SIF programs, the development of a unique universal identifier
for all SIF participants to facilitate
data management at the local and portfolio level of evaluation
and the creation of a data
management system and protocol for all SIF programs at the
portfolio level.
We have used the model developed in the SIF program to manage
and support evaluation efforts in
other initiatives. The basis for the RFQ in our Evolved ELC
initiative rested in part on an evaluation
conducted by JFM Consulting. Those evaluation results, coupled
with what we had learned in the
execution of the SIF model, led to a tightening of the logic
model/theory of change, a reallocation of
resources, a reduction in number of partner agencies and a
reinvigorated collaborative decision-
making structure. The on-going evaluation of the Help Me Grow
Initiative, by Oakland University,
has scaled efforts toward proven successful strategies.
In addition to incentivizing agency partners through the
RFP/RFA/RFQ process, UWSEM has
continually moved forward in building our capacity and policy of
providing technical support and
For Official Use Only
Page 18
-
Narratives
resources to facilitate program partners to increase their own
capacity for and organizational culture
toward rigorous evaluation. For example, UWSEM has invested in
an improved data system for its
Early Childhood work. Internal and partner staff have been
trained in Social Solutions' Efforts to
Outcomes database to collect and analyze data in order to make
efficient resource allocation decisions,
improving program performance.
Ten years ago UWSEM began an organizational cultural and
practice shift toward evidence based
decision making and data informed organizational strategy. This
process has culminated in a recent
reframing of organizational strategy throughout the entire
organization highlighting the importance
of applying evidence and evaluation results to decision
making.
UWSEM also has developed an entire unit focused on evaluation
and research relating to UWSEM
programs. The organization has committed to continuing to build
capacity to use performance and
results data to assess strategies, measure results and engage in
ongoing improvement efforts for
UWSEM initiatives and partnerships.
UWSEM has worked extensively with contractors to facilitate
capacity building both internal to the
organization and with all of our program partners for the
successful implementation and utilization of
rigorous evaluation designs and findings. Contractors include
Child Trends, Data Driven Detroit,
Here2There, Ty Partridge Ph.D, and JFM Consulting.
Under our Strategy B plan, we will lead the evaluation effort of
the BtoB SIF model and evaluation
design detailed below. A critical part of this plan will be
assessing sub-recipient applicants for readiness
and capacity to fully engage and support this effort. Our
scoring rubric for evaluating potential
program partners will include their ability to implement
evidence-based interventions with fidelity and
to collect, store and use data at a level consistent with an
evaluation plan that can achieve moderate
or strong evidence over a three-to-four year time period.
Since we will be leading the single evaluation plan, we envision
the technical assistance needed will
be focused on ensuring that partners understand the plan and are
able to support the data collection
needed. Our selection process is rigorous enough to make this
determination going in. We will draw
on the SIF model of regular Learning Community meetings and the
ELC model of providing data
collection TA. Our RFQ will include descriptions of the role of
staff. Our experience in the Evolved
ELC RFQ supports this approach.
ASSESSING EVALUATION CAPACITY and TECHNICAL ASSISTANCE: We
intend to use the
first six months of the grant period to conduct a comprehensive
evaluability assessment to identify,
develop and appraise evaluation documentation and assessment
procedures, implementation logistics,
For Official Use Only
Page 19
-
Narratives
data management protocols, and organizational capacity in terms
of staffing, training, technological
infrastructure, and other evaluation related resources.
This is an essential part of the evaluation process as it
identifies on the front end, prior to
participant enrollment and data collection, that the evaluation
process can be effectively implemented
and has the capacity to yield high quality information that
captures program implementation as well
as program outcomes and impact. The evaluability assessment will
be comprised of at least 4 core
elements: (1) Assessment of partner agencies, (2) development of
a data model, (3) development of a
data collection and management procedure, and (4) specification
of specific evaluation questions at all
phases of evaluation (i.e., implementation, outcome, and impact)
and analysis strategies which are
isomorphic to each evaluation question.
UWSEM will develop a scoring rubric through which partner
agencies will be assessed for their
readiness to participate in a rigorous evaluation process. The
rubric will include, but not be limited to:
(1) Agency experience implementing and utilizing high quality
evaluations in their work; (2) a
demonstrable organizational culture which embraces evaluation
and uses a data driven strategy; (3)
the technical/technological capacity for the collection and
management of evaluation data; (4) the
agency's staffing and budgetary capacity to be able to
participate in the BtoB SIF evaluation and meet
the expected standards of rigor.
UWSEM will base partnering and sub-grantee decisions in part on
the agency's evaluation
readiness score using this rubric. UWSEM will also provide
technical assistance to agencies that play a
critical role in program implementation and have adequate
evaluation capacity, but would benefit
from increasing their evaluation capabilities.
UWSEM will develop and document a comprehensive data map which
will identify each source of
information required to accurately assess program
implementation, potential barriers to
implementation, program outcomes, and program impacts. Each
information source will be
operationally defined and a quantitative indicator(s) will be
identified. All data elements will be
explicitly linked to the logic model and theory of change.
Further each data element will include a
description of the data source, responsibility for collection,
and the procedures for obtaining,
managing, and accessing the data. In the case of evaluation
outcomes and impacts which employ a
psychometric assessment tool, a full description of the measure,
reliability and validity information,
and a verification that the measure meets or exceeds acceptable
scientific criteria for use (e.g.,
Chronbach's alpha >.7 and demonstrated construct validity).
Drawing on the data model,
implementation, outcome and impact evaluation questions will be
operationally specified.
For Official Use Only
Page 20
-
Narratives
As a final step in this evaluability assessment, UWSEM will
develop a training protocol to be
delivered to all evaluation project staff at UWSEM and partner
agencies. This will be a comprehensive
training focusing on evaluation design, data collection and data
management protocols. UWSEM will
also implement "booster" trainings on a regular basis (at least
quarterly). The purpose of these booster
trainings will be to obtain feedback regarding effective
evaluation related activities, barriers and/or
challenges to the evaluation process, and to ensure that any
drift in the fidelity of evaluation
procedures is minimized.
Following the six-month evaluability assessment phase of the
project, UWSEM will initiate an
evaluation implementation pilot phase. Evaluation procedures
will be implemented on a randomly
selected group of participants with a target sample size of
n=40. This will allow for a thorough
assessment of the evaluation process including, data collection,
data management, and
methodological protocols prior to implementing the evaluation on
a large scale basis. The piloting of
the evaluation process is intended to identify any barriers or
challenges to the integrity of the
evaluation and address those issues with minimal impact on the
quality of the evaluation findings.
The assessment of the pilot evaluation will involve a complete
review of all aspects of the evaluation
and the use of key informant interviews (frontline staff,
program leads, program participants, data
collection and management staff, and all other staff involved in
the evaluation process). The
procedures developed to conduct this review will be maintained
on a smaller scale throughout the
course of the entire evaluation to ensure timely feedback
between all key individuals involved in the
evaluation process for continuous quality control.
In addition, special focus will be given to assessing the
efficacy of program recruitment and
retention procedures and the adequacy of any propensity and/or
case-control matching procedures.
Moreover, an initial data entry auditing process will be
implemented in which no less than 30% of the
pilot cases will be double entered and checked for consistency.
Any data entry discrepancies will be
compared to the original raw data source and corrected. If there
appear to be systemic factors which
made data entry errors more likely the UWSEM evaluation team
will adjust the data collection
protocol in order to resolve these issues. If the data entry
error rate exceeds 10% a regular auditing of
20% of all evaluation cases will be implemented throughout the
evaluation process in order to
continue to ensure the highest quality data accuracy possible
without incurring the cost of complete
double entry methods.
Beginning with year 2 of the BtoB SIF project UWSEM will begin
to implement the full scale
evaluation. The detailed evaluation plan will be further
developed and refined during the first year
For Official Use Only
Page 21
-
Narratives
evaluability and piloting phase of the project. However, we have
outlined a core set of evaluation
requirements which represent the minimum standards for the
evaluation plan.
ACHIEVING MODERATE EVIDENCE with STRATEGY B: Because of the
nature of the proposed
intervention program the evaluation design will inherently
involve a longitudinal design. A
longitudinal design has a number of key strengths for
elucidating evaluation outcomes. Specifically,
the use of longitudinal designs will allow for the assessment of
intra-individual variation which is a
necessity when the research aim is to uncover the impact of
intervention effects on developmental
trajectories. The proposed intervention model is heavily
informed by contemporary developmental
science and the key unit of analysis is the process of change
within the individual over time. Indeed,
what we are most focused on in this evaluation is the prediction
and ability to change the course of
development within individuals and families as opposed to mean
scores on a single measure at a given
point in time.
In addition, within the context of the longitudinal design we
will employ a baseline assessment
with multiple post-intervention assessments. This design feature
controls for a number of threats to
internal validity when comparing baseline and post-intervention
differences. It also increases the
statistical power of the evaluation.
The key evaluation research question concerns the degree to
which the intervention alters the
developmental trajectory of "at-risk" children and families in
comparison to the absence of the
intervention. The longitudinal/repeated measures design will
allow for the assessment of trajectories
of change, however, in order to fully address this key research
question a comparison group which is
also assessed longitudinally is necessary. Complete random
assignment is the gold standard design and
has the greatest degree of efficiency for drawing causal
inferences. However, because we are using a
"no wrong door" approach to program enrollment a true random
assignment design is impractical
from a logistical standpoint. Moreover, the comprehensive
ecological nature of the proposed
intervention further restricts the feasibility of a employing a
randomized control group. As such we
propose to use a propensity scoring method from which we can
draw a comparable control group
from a sample of children and families served by UWSEM program
partners who are not directly
involved in BtoB SIF.
The use of propensity scoring methods has been expanding in
program evaluation as it provides an
alternative to randomized control trials, which are often
untenable to implement (Guo & Fraser,
2013). Propensity scores are conditional probability estimates
of receiving treatment, given a set of
measured covariates. This can be expressed as a binary logistic
regression, where P is the probability of
For Official Use Only
Page 22
-
Narratives
person i receiving treatment T, given that person i has a
specific set of values x on the vector of
conditioning or covariate variables X. These covariate variables
X are potential confounds related to
self-selecting into treatment, which might be an alternative
explanation for mean differences in the
outcome for those receiving treatment and those in a control
group.
Using this approach, an individual propensity score can be
calculated in the form of a logit, or log
odds, estimating the likelihood of receiving treatment--given
the values of the covariates for that
person. Once these logits are calculated, individuals in the
treatment group and the control group are
matched based on their propensity scores, or their individual
logit values. Since these values are
usually not exactly the same, a method is needed to determine
how close a control propensity score
needs to be to a treatment propensity score in order to be
considered a match.
There are a number of methods for estimating matches, two of the
most common being nearest
neighbor-matching and calculating Mahalanobis d (i.e. distance).
Mahalanobis d is commonly used in
multivariate statistics to estimate the average difference in
scores on a set of variables across
participants. The method has been adapted to propensity
score-matching as well.
We will use the Mahalanobis d method because of the potentially
large number of variables across
which we will want to match intervention and control
participants. Propensity scores will be
calculated based on the following covariates: family income, age
of parents, age of participating child,
number of other support programs in which the family
participates, parent education, primary
language used in the home, ethnicity, and other variables
identified during the evaluability and pilot
phases.
In addition we will take advantage of two features of the
proposed intervention strategy which will
likely create natural control groups. The intervention model
targets families during the prenatal and
neonatal periods and also incorporates a routine check-up
intervention in which families are regularly
contacted and can be assessed. We anticipate that not all
potential program participants will utilize the
program during the initial contact or even follow-up contacts.
However, we intend to assess those
families to the extent possible thus providing a quasi-waitlist
control group.
Baseline equivalence: As a result of the no wrong door policy
for program entry a randomized
control group cannot be feasibly implemented. Further, there
will be multiple "baseline" conditions
depending upon the manner and timing in which a family enters
the program. As such, we will
rigorously assess families in the intervention and in the
selected control groups to determine
equivalence at baseline entry points. All key measures will be
compared between the two groups using
an independent sample t-test using an a priori alpha
significance criterion of alpha=.10 in order to
For Official Use Only
Page 23
http:alpha=.10
-
Narratives
obtain a conservative estimate in the context of a large sample
size. Any variables on which the
intervention and control groups are found to differ at the p=.10
level will be included as baseline
controls in all subsequent evaluation analyses comparing the two
groups.
External validity: One of the central aims of the proposed BtoB
SIF initiative is to bring the
intervention to scale to at least a regional scale. It is
important therefore to include an explicit
assessment of the external validity of the findings. UWSEM will
develop a regional demographic
model using a variety of data sets obtained through 2010 census
data, Data Driven Detroit, and Kurt
Metzger and Associates. Using this demographic model an
assessment tool will be developed to use
with all program participants. This measurement tool will allow
for the direct comparison of the
intervention sample to the larger regional population.
Organizational Capability
Sample Size, Statistical Power, and Effect Size Considerations:
Through the current SIF project
UWSEM has served, through our SIF sub-grantees, 19,000 children
and families. Additionally,
through the Help Me Grow project, UWSEM has gathered data on
over 40,000 children aged 0-3 in
the Metro-area and approximately 95,000 early childhood
assessments using the Ages and Stages
Questionnaire. Since inception, UWSEM has provided services to
approximately 29,000 adults,
through our ELC networks. As such, UWSEM has developed the
capacity to recruit and assess
relatively large samples drawn from the same "at-risk"
population which we are aiming to serve in the
current proposal.
Based on our working history with the currently operating
components of the proposed
intervention we anticipate enrolling approximately 1,200 youth
and families drawn from
approximately 12,000 screened families as well as recruiting a
control group of a minimum of 800
families across the four year intervention period. In addition
we intend to get a minimum of three
assessments of all outcome measures for each family. Assuming an
80% retention rate across the
three assessment periods we will have a base observation sample
of 2,880 observations. This will yield
more than adequate statistical power for all proposed
statistical models. However, for each proposed
analysis a detailed power analysis will be conducted. For
traditional general linear model analyses
(e.g., ANOVA, MANOVA, regression analyses) statistical power
will be assessed using G*power v. 3.1.
For more advanced statistical modeling (e.g., Latent Growth
Models, Structural Equation Models,
Latent Class Models, etc.) statistical power will be estimated
using the Monte Carlo feature of Mplus v.
7.1. This latter method allows for the specification of the
statistical model to be tested with minimally
acceptable parameter estimates. These fully specified models are
then simulated using Monte Carlo
For Official Use Only
Page 24
-
Narratives
methods across 1,000 simulated samples. The proportion of
samples with which the path coefficients
are determined to be statistically significant at the p=.05
level are generated and if the proportion is
greater than 80% with a specified sample size, then it is
determined the model has adequate statistical
power.
In addition, for all structural equation based models we will
calculate power at the model level
using the root mean square error of approximation (RMSEA) fit
index as the effect estimate
epsilonfor; epsilon=0 (exact fit), epsilon less than or equal to
.05 (close fit), and epsilon > .05 (poor
fit). This approach estimates the likelihood of the value of
epsilon; under the null and alternative
hypotheses for a given sample size N, significance level alpha;
and model degrees of freedom (ratio of
observed variables to estimated parameters).
All statistical power estimates will assume alpha=.05, power
=.80, and a moderate effect size of d =
.35. Generally a Cohen's d of d = .2 is considered a small
effect size and d = .5 a medium effect size.
Effect is of particular importance when testing statistical
models in large samples because significance
levels are exponentially proportional to sample size such that
as sample sizes increase, smaller effects
are determined to be statistically significant. By explicitly
reporting the effect size, the results can be
interpreted in terms of meaningfulness as opposed to statistical
significance. This is particularly
relevant for applied evaluation research which has the aim of
informing policy and practice. In this
context reporting very small effects to be statistically
significant resulting from a large sample size will
likely result in the inefficient use of money and resources to
produce a very small impact. As such, we
will not interpret findings with an effect size smaller than
d=.35 which is considered to be in between a
small and medium effect.
Missing Data: All evaluation analyses will use an
intent-to-treat analysis approach and incorporate
all participants in both intervention and control groups
irrespective of program adherence and /or
missing assessments. Intent-to-treat analyses correct for the
potential overestimation of treatment or
intervention effects which result from removing non-adhering
participants from the sample.
In addition, prior to analysis all data will be assessed for
patterns of missingness in order to
determine the likelihood that missing data are; (1) Missing
Completely At Random (MCAR), (2)
Missing At Random (MAR); or (3) Missing Not At Random (MNAR). If
at least MAR a combination
of multiple imputation techniques and the use of
Full-Information Maximum Likelihood estimates
will be employed depending on the type of analysis being
conducted. Under conditions of MNAR,
patterns of missingness will be included in all statistical
models as a control.
Analysis strategies - Implementation and Program Fidelity
Analyses: The complete set of
For Official Use Only
Page 25
http:alpha=.05
-
Narratives
evaluation research questions will be specified and refined
during the evaluability and pilot phases of
the project. However, there is a core set of general evaluation
questions which will be required as part
of the final evaluation.
All key program components will be tracked and evaluated to
assess the degree to which the fidelity
of the intervention model was upheld and the degree to which the
intervention was fully implemented
across participants. Fidelity and implementation questions will
include: adequacy of training for
intervention staff, degree of adherence to intervention
protocols, number of children and families
served, types of services provided, and number of referrals to
partner agencies.
In addition, program participation variables will be assessed
and incorporated as moderators in all
outcome and impact analyses. Because UWSEM is adopting a "no
wrong door" program engagement
policy it will be important to identify participants
characterized as early adopters (i.e., prenatal,
neonatal, early infancy), middle adopters (late infancy thru
toddlerhood), and late adopters (preschool
-- K). Evaluation analyses will utilize discriminant function
analysis to identify clusters of baseline and
demographic variables which predict adopter status (i.e., early,
middle, late). Adopter status will also
be included as a moderating variable for analyses of program
outcomes and impacts.
Similarly, Cox regression hazard models will be conducted to
predict the likelihood of intervention
duration or the length of time a participant stays engaged with
the intervention. Baseline and
demographic variables will also be included as predictors of
intervention duration. Hazard models are
required for these types of analyses because incorporating a
unit of time as an outcome variable
violates several core assumptions of inferential statistics
(David 1972, Dickman, Sloggett et al. 2004).
Time is a monotonically increasing variable and thus cannot be
treated as a random variable with a
normal distribution and independence of observation.
Additionally, there will be individuals for whom
intervention participation does not end before the end of
evaluation data collection. This creates an
inestimable probability distribution referred to as right data
censoring which severely biases traditional
statistical tests. Survival or hazard models are explicitly
designed to address these two features of
"time-to-event" data such as intervention duration.
The level of intervention engagement will also be assessed.
Intervention engagement will be
operationally defined as: (1) the ratio of number of
intervention services for which a participant is
eligible to the number of intervention services received; (2)
the number of intervention components a
participant utilizes; and (3) a self-report index of the degree
to which intervention participation is felt
to be an important part of the participant's life. A composite
engagement variable will be created from
these three indice (Whether this is a latent construct or a
weighted linear sum will be determined
For Official Use Only
Page 26
-
Narratives
during the pilot phase of the evaluation) and used in all
outcome and impact analyses as either a
predictor or as a moderator.
Full Intervention Model v. Intervention Components: The proposed
intervention model (see
LM/TOC) is a multifaceted intervention with multiple
intervention components. It is anticipated that
some participants will engaged in some intervention components
but not others. Indeed, the adoption
of a "no wrong door" intervention access policy ensures
differential intervention utilization patterns to
some degree. To the extent possible, the evaluation strategy
will attempt to capitalize on this aspect of
the intervention and evaluate intervention outcomes for both the
entire intervention model as a whole
as well as individual components of the intervention as
quasi-standalone interventions. This will allow
for the comparative efficacy of the integrated intervention
model relative to individual intervention
components.
Outcome and Impact Analyses: As discussed above, the
specification of specific evaluation research
questions, especially with respect to impact and outcome
analyses is premature at this point and will
be fully developed during the evaluability and pilot phases of
the evaluation. However, as with the
implementation and fidelity evaluation there are several
analytic strategies which will be necessary
requirements for the final evaluation plan.
Multilevel Models: The data collected as a part of this
evaluation plan will inherently be nested or
non-independent on a number of levels of analysis. Because of
the longitudinal design the repeated
observations over time within each participant will by
definition be non-independent. Additionally,
because components of the intervention will be delivered in a
group context it is assumed that
participants within a given group will have non-independent
data. There are also potential non-
independence effects due to factors such as neighborhood,
classroom, hospital setting, etc. In cases
where data could potentially have a nested or non-independent
structure intra-class correlations
(ICC) will be estimated for all salient variables. The ICC is an
index of non-independence of
observation ranging from 0 (independent) to 1 (complete
dependence). In cases where the ICC is
statistically significant, indicating non-independence all
analyses will be conducted using a multilevel
modeling framework. Where level 1 represents the relationship
between the independent and
dependent variable and level 2 disaggregates the independent
relationship of the independent and
dependent variable and the variance due to non-independence of
observation such as time nested
within person or person nested within group. These models yield
unbiased estimates of relationships
between independent and dependent variables, as well as allowing
for the explicit estimation of group
level predictors. This multilevel framework has also been
extended beyond the regression based
For Official Use Only
Page 27
-
Narratives
framework to latent variable model frameworks as well.
Latent variable structural equation modeling (LVSEM): One of the
key evaluation questions is to
test an ecological model linking social determinants of family
risk to child outcomes through a
number of mediating child and family process variables. Social
determinants will include; relative
poverty as measured by the Arizona Self-Sufficiency Matrix
(ASSM), barriers to health care access,
and cumulative family risks. Child outcomes will include
assessments of the five core areas of child
health and well-being: (1) language and communication; (2) gross
and fine motor skills; (3) general
cognitive skills; (4) socioemotional well-being; and (5)
physical health and nutrition. A
comprehensive set of mediating variables which are directly
linked to the intervention targets in the
proposed intervention model will be assessed as well. The
selection of these variables will be drawn
from an extensive review of the infant and early-childhood risk
and resilience literature and will
include factors such as: Caregiver health and emotional
well-being, caregiver social support,
knowledge and use of positive parenting practices, parent-child
relationship quality, access to and
utilization of high quality childcare, home supports for child
development. Further, important
moderating factors including; caregiver race/ethnicity,
community risk factors, and the economic,
health, and family policy contexts will also be considered.
In order to fully evaluate these process models latent variable
structural equation models will be
employed (MacCallum and Austin 2000, Tomarken and Waller 2005,
Weston and Gore 2006, Little,
Card et al. 2007). These models have a number of advantages over
regression based models. First,
they allow for more precise measurement models of key constructs
by estimating latent variables
across multiple observed variables which are considered to
measure the same underlying construct.
These latent variables can be represented in matrix form as a
series of weighted linear sums generally
referred to as a measurement model. The second key advantage of
LVSEM is that it allows for the
specification of complex relationships between latent constructs
through simultaneous estimation of
structural (i.e., regression) equations. This structural part of
the model can also be represented in
matrix form. Further, the path coefficients in the model can be
decomposed into direct and indirect
effects to provide formal tests of mediation processes.
In order to test categorical moderating factors at the
statistical model level, multi-group LVSEM
will be used in which a particular SEM will be specified for
each level of the moderating variable (e.g.,
ethnicity) and all path coefficients will be constrained to be
equal across moderator levels. Overall
model fit in this constrained model will be assessed and a poor
fitting model will suggest moderation
effects. Subsequently, the path coefficient constraints will be
removed across levels of the moderator
For Official Use Only
Page 28
-
Narratives
allowing each level to produce different path coefficient
estimates which can then be compared to
better interpret the distinct statistical processes for each
level of the moderator.
Mixture model latent growth modeling: Another key evaluation
question will be to determine the
process of change in a number of key index variables including
relative poverty and child outcomes.
Of particular interest is the assessment of trajectories of
change in these variables over time as
opposed to a single point-in-time assessment. By using latent
growth modeling (LGM), which is a
special case of LVSEM, intra-individual variance or growth
trajectories can be estimated for each
participant and predictors of variability (e.g., treatment v.
control) can be incorporated (Muthén and
Muthén 2000, Li, Duncan et al. 2001, Duncan and Duncan 2004).
These models are an integration of
the multilevel models and LVSEM approaches described above in
which the latent variables are
comprised of two weighted linear sums of individual scores on a
given measure over time referred to
as the latent intercept and the latent slope. The means of the
intercept and slope are the sample
average for a measure at a given point in time(Intercept) and
the average rate of change in that
measure over time (slope). The variance of these two latent
variables provides an index of inter-
individual variance in intra-individual variability over time
(Nesselroade & Ram, 2004; Willett,
2004). As such, change in a variable over time is explicitly
incorporated and can be both predicted by
and predictive of other salient variables in the model.
Mixture models or latent class models are a special case of
latent growth models in which latent
classes C are estimated from the variance of the latent
intercept and slope and represent distinct
statistical populations based on the overall level and rate of
change in the outcome variables over time
(Muthén and Muthén 2000). So for example, a class may be
comprised of individuals that have
relatively high initial levels of the outcome variable and show
rapid reductions over time whereas
another class may be comprised of relatively high initial levels
but show little to no change in outcome
over time. These latent classes can be used to identify
participants that could be categorized as "high
responders", "moderate responders" and "low responders" to the
intervention. These latent classes of
"responder" can then be predicted by baseline, demographic or
intervention engagement/utilization
variables thus informing optimal target populations or the need
to adjust the intervention for some
groups of participants based on the response likelihoods.
STRATEGY B -- CURRENT AND POTENTIAL LEVELS OF EVIDENCE:
Family Check-Up Model: The family checkup model is a
motivational interviewing base