0 The Children’s Aid Society Twenty-first Century Community Learning Center Grant (2010-2013) Evaluation Report Prepared by Eoin Collins, Muamer Rasic and Heléne Clark December, 2013 ActKnowledge 365 Fifth Avenue, 6th Floor New York, NY 10016 Telephone 212.817.1906 www.actknowledge.org
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
This is a report of the results of an evaluation focused on the two Children’s Aid Society
community schools in New York City where the elements of the community school model are at
an advanced stage of implementation. These are:
Salomé Ureña de Henríquez Community School, which incorporates three middle
schools on one campus (I.S. 218, M.S. 293, and M.S. 322); and
Mirabal Sisters Community School, which incorporates two middle schools on one
campus (M.S. 319 and M.S. 324).
Both schools are based in Washington Heights and have been community schools for some
time. They were selected by CAS and the evaluators as the best implementers of the model out
of the five community schools funded for the last five years under 21st Century Community
Learning Center grants. The previous years of evaluation looked at implementation and
outcomes in all five schools. As one would hypothesize, outcomes were best when
implementation was strongest. Therefore, this evaluation focuses on outcomes and lessons in
schools in which implementation has the most fidelity to the CAS Community School model.
The evaluation focused in particular on the progress made by these two schools over the period
2010-20121 in achieving outcomes linked to various preconditions for student success
articulated in the Children’s Aid Society Community Schools Theory of Change (ToC). Key
findings on youth outcomes and parent/family engagement are summarized as follows.
Youth Outcomes
Academic Achievement
Average student proficiency in Math in both community schools increased each year
from 2010 to 2012. Proficiency increased among after school participants and students
in each school as a whole. Over this period Math proficiency of after school students
increased from 46% to 57% compared to school wide improvements of 39% to 55% for
Salomé and of 52% to 59% for Mirabal.
Average proficiency in English Language Arts (ELA) for each school has been lower and
somewhat more variable than proficiency in Math between 2010 and 2012. This is
correlated to the significant number of English Language Learners who comprise over
40% of students in both community schools; in the school with fewer ELL students, ELA
proficiency was significantly higher.
1 DOE instituted a new test in the 2009-2010 school year, and again in the 2012-2013 school year to reflect Common Core curricula. Therefore, the years available for longitudinal comparison are 2010, 2011 and 2012.
CAS 21st Century CLC Grant Evaluation 2010-2013
3
Tests changed between 2012 and 2013 and therefore it is not possible to compare 2013
results to previous years. Nationally, and within CAS community schools, scores on the
new test trend lower. Future tests will have to use the 2013 test as a baseline.
Attendance and Absentee Rates
The leadership of the Children’s Aid Society in understanding and addressing
absenteeism and chronic absenteeism is having an effect in the two schools under
review.
Both community schools have had attendance rates above 90% through 2010-2013,
broadly in line with the average attendance rates for middle schools in the city through-
out this period.
Chronic and severely chronic absentee rates in both schools have decreased from 2010-
11 to 2011-12 academic school years and the rates were lower than the city average for
2011-2012 academic school year.
Youth development
A significant majority of after school students were very positive about the after school
program in their school believing it to be a comfortable and supportive environment.
The CAS after school students are participating in a variety of academic and enrichment
programming ranging from use of computers to physical and recreational activities.
Surveys indicate that the after school programs are having a positive impact on social
and emotional development. For example, a majority of students from 2010-2012
agreed that the program in their school is providing positive role models and helping
them get on better with others.
The surveys also indicate the after school program activities are helping participants
improve reading, writing, and math and develop the capacity and skill to do better in
school.
Afterschool participants score highly on questions relating to goals and aspirations. For
example, more than 90 percent of after school students in both schools indicated an intent to
attend and finish college.
Parent/Family Engagement
The two community schools scored higher than the city average on every domain on
parent/family engagement measured in Department of Education parent surveys from 2010 to
2013. In particular:
CAS 21st Century CLC Grant Evaluation 2010-2013
4
The response rates to the DOE Parent Surveys, which can be considered an indicator of
parent engagement, are higher for both community schools than the city average.
The schools also scored higher than the city average on all domains including ‘safety and
respect’, ‘academic expectations’, ‘communication’ and ‘engagement’ by the school.
CAS 21st Century CLC Grant Evaluation 2010-2013
5
1. Introduction and Background
This is a report of the results of a 21st Century Community Learning Center (21CCLC) evaluation
focused on the two Children’s Aid Society community schools in New York City where the
elements of the CAS community school model are at an advanced stage of implementation.
These schools are Salomé Ureña de Henríquez Community School and Mirabal Sisters
Community School.
The evaluation of the schools has been carried out with reference to the Theory of Change for
Community Schools developed by CAS and ActKnowledge, which sets forth the range of
preconditions necessary to achieve overall student success. This encompasses preconditions
around youth development, academic achievement, attendance and participation,
parent/family engagement, and involvement with the schools.
In keeping with the goals of 21st Century Community Learning Center (21CCLC) funding, the
successes of programs funded by 21CCLC, in particular the after-school program, are the basis
of analysis. Students in 21CCLC after-school programs are compared with students not
attending, and school-wide results are reported as well.
1.1 Background to the Evaluation
This is the fifth year of a 21st Century Community Learning Center Grant which The Children’s
Aid Society received for programs at five schools –Salomé Ureña de Henríquez Campus, Mirabal
Sisters Campus, Vito Marcantonio P.S./I.S. 50, The Bilingual Magnet School P.S.211, and
Herman Ridder I.S. 98. For the following reasons, CAS and the evaluators decided that more
could be learned from examining the schools with highest fidelity (Salomé Urena de Henriquez
and Mirabel Sisters Campus) to the CAS Community School model. These factors determined
the focus of the Year 5 evaluation:
1. Previous evaluation has confirmed, as hypothesized, that better implementation leads
to better outcomes.
2. Conversely, reporting on outcomes for schools that have challenges and barriers to
integrating and providing services does not shed light on whether the community school
model is, itself, effective.
3. The two schools with the most success at implementation have the most to teach us
about the connection between program offerings, student and school need, and
achieving outcomes.
CAS 21st Century CLC Grant Evaluation 2010-2013
6
4. On a practical level, evaluation resources were lower in Year 5, and priority was given to
drawing lessons about what happens when programming is actually implemented as
intended.
1.2 Methods
The evaluation used a mixed-methods approach of qualitative and quantitative analysis. These
include:
Development and Refining of the Theory of Change
The Children’s Aid Society and ActKnowledge developed a comprehensive Theory of Change
for Community Schools in 2011. It is a good model of Community School components and it
serves as a basis for evaluation. The theory is described in detail in Section 3.
Site Visits
ActKnowledge conducted site visits at each community school using interview protocols,
focus group protocols, and observational protocols designed to elicit the views of
stakeholders on developments and achievements of community school programming. Visits
at each site involved structured meetings and interviews with stakeholders including:
o Interviews with community schools site directors
o Interviews with principals
o Interviews with parent coordinators
o Interviews with after school staff
o Focus groups with parents
o Focus groups with students participating in the after school program
o Observations of after school activities.
Youth Surveys
ActKnowledge analyzed results from youth surveys which have been developed and
conducted each year by The Children’s Aid Society and which are posted on
youthservices.net. Note: End of school year surveys were not undertaken in 2013 so the
results in the evaluation are for the years 2010-2011 and 2011-2012.
Identification and Analysis of Academic and Attendance Data
ActKnowledge analyzed academic and attendance data from youthservices.net and the New
York City Department of Education.
o The academic data were analyzed for the 2009-2010 through 2011-2012 academic
years, and findings were reported on ‘average % proficiency on Math and ELA’ State
Exams for Salomé Ureña Campus, Mirabal Sisters Campus, and CAS after school
CAS 21st Century CLC Grant Evaluation 2010-2013
7
participants in general. DOE instituted a new test in the 2009-2010 school year, and
again in the 2012-2013 school year to reflect Common Core curricula. Therefore, the
years available for longitudinal comparison are 2010, 2011 and 2012.
o The attendance rates were analyzed from 2009-2010 through 2012-2013 academic
school years.
o The chronic and severely chronic absentee rates were analyzed from 2010-2011
through 2011-2012 academic school years on Salomé Ureña Campus, and Mirabal
Sisters Campus, and a comparison to the citywide average was conducted for the
2011-2012 academic school year.
Parent Surveys
ActKnowledge analyzed results from Department of Education Parent Surveys from 2010-
2013 on Salomé Ureña Campus and Mirabal Sisters Campus and compared the results with
the citywide average. The Department of Education parent survey measures the perception
and satisfaction of parents with their children’s schools across domains that include ‘safety
and respect’, ‘academic expectations’, ‘communication’, and ‘engagement’.
CAS 21st Century CLC Grant Evaluation 2010-2013
8
2. Profile of the Community Schools and Community School Theory of
Change
2.1 Salomé Ureña de Henriquez Campus (I.S. 218, M.S. 293, M.S. 322)
The Salomé Ureña Middle Academies (SUMA IS 218) was established as a Full Service
Community School in 1992 through a partnership between The Children’s Aid Society, the New
York City Department of Education, Community School District Six, and community-based
partners. Located in Washington Heights, the school was purpose-built as a community school
and was, with P.S. 5, among the first community schools to be established by CAS.
In 2004, Salomé Ureña Middle Academies was divided into three schools: I.S. 218, M.S. 293
(now called City College Academy of Arts) and M.S. 322. The Children's Aid Society still provides
full services and programs to all schools on the campus. Services and programs include after-
school, Saturday, holiday and summer programs. They also include mental health, medical, and
dental services; and a set of services relating to family support and community engagement.
These include a family resource room, vocational and educational training, adult education,
advocacy and leadership opportunities. From its inception the school has served as a prototype
for the community schools model for hundreds of community schools in the United States and
abroad.
Chart 1 below shows enrollment for each constituent school in Salomé Ureña. The total number
of students that attended Salomé Ureña Campus was 997 in 2012-2013.
Chart 1: Enrollment at Salomé Ureña Campus
As noted in Chart 2, English Language Learners have comprised a significant part of the student
body in I.S. 218 and M.S. 322 - ELL students comprised 45.6 percent of students in I.S. 218 and
40.4 percent of students in M.S. 322 in 2012-2013. The City College Academy of Arts (CCAA) has
significantly fewer English Language Learners.
368 365 341 309
490 494 463
428
258 270 271 260
0
100
200
300
400
500
600
2009-10 2010-11 2011-12 2012-13
IS 218
MS 322
CCAA 293
CAS 21st Century CLC Grant Evaluation 2010-2013
9
Chart 2: % ELL Students at Salomé Ureña Campus
2.2 Mirabal Sisters Campus (M.S. 319, M.S. 324)
The Mirabal Sisters Campus opened in 1994 as I.S. 90 in collaboration with The Children’s Aid
Society and in 2004 was divided into three schools, two of which subsequently merged.
The Children's Aid Society provides similar services and programs to those which have been
developed in Salomé Ureña Campus including an after school program, various health services
and services to support parents and the wider community. The Mirabal Sisters Campus is also
located in the Washington Heights/Inwood neighborhood of Manhattan.
Chart 3 below shows the number of enrollments from 2009 to 2013 with a total enrollment of
1054 in 2013. As illustrated in Chart 4, Mirabal also, like Salomé, has a significant number of
English Language Learners comprising 40.1 percent of students in M.S. 324, and 35.3 percent at
M.S. 319 in 2013.
Chart 3: Enrollment at Mirabal Sisters Campus
44% 45.5% 44.3% 45.6%
36.3% 37.7% 38.2% 40.4%
11.6% 12.6% 9.2% 7.3%
0%
10%
20%
30%
40%
50%
2009-10 2010-11 2011-12 2012-13
IS 218
MS 322
CCAA 293
538 570 649 634
422 407 427 424
0
100
200
300
400
500
600
700
2009-10 2010-11 2011-12 2012-13
MS 319
MS 324
CAS 21st Century CLC Grant Evaluation 2010-2013
10
Chart 4: % ELL Students at Mirabal Sisters Campus
2.3 Children’s Aid Society Community Schools Theory of Change
The Children’s Aid Society (CAS) and ActKnowledge developed the Community Schools Theory
of Change over the course of a year and completed it in May 2011. The CAS Community Schools
Theory of Change was created in a series of participatory meetings and is the basis for CAS
evaluation efforts. It serves as a fairly good model of Community School components in general.
The theory is a good starting point for any initiative, although each should reflect its particular
context, needs and priorities.
The CAS Community Schools Theory of Change has three outcome pathways to achieve its long-
term outcome of “Youth have opportunities for positive engagement after high school.” The
three outcome pathways consist of Youth Outcomes, Parents and Families Outcomes, and
Institutional Outcomes.
Youth Outcomes Pathway
A key precondition to the long-term outcome ‘Youth have opportunities for positive
engagement after high school’ as articulated in the Theory of Change is ‘Young people succeed
academically’ – i.e., students are earning good grades in class, passing state exams, and
graduating. For young people to succeed academically they have to have ‘high attendance.’
They must be present in school to keep up with class materials and homework. A precondition
to regular attendance is good health – i.e., students are not out sick on a regular basis. Also, a
‘school-based health center’ is there to provide ‘quality health care’ so that students can be
treated in the school rather than be out sick.
Parent and Family Outcomes Pathway
The Children’s Aid Society has identified parent and family engagement as an important
precondition to youth outcomes as articulated in the Theory of Change. The long-term outcome
36.8% 38.2% 39.1%
35.3%
40.0%
44.5% 44.0%
40.1%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
2009-10 2010-11 2011-12 2012-13
MS 319
MS 324
CAS 21st Century CLC Grant Evaluation 2010-2013
11
of the parents and family pathway is ‘Parents/Families are partners:’ They become partners by
‘engaging at every level’– i.e., by supporting their children’s academic needs at home, creating
an ‘in-home learning’ environment, being supportive at school, being ‘active participants at
school,’ by attending school events and meetings, and so on.
Institutional Outcomes Pathway
The Institutional Outcomes pathway shows short-term preconditions to outcomes in the parent
and family pathway as well as the youth pathway. Institutional outcomes (preconditions)
comprise ‘securing sustainable funding,’ ‘sharing data, results, and evidence with partners,’
‘capacity building at the site level,’ and ‘strong and effective leadership;’ all of which make the
‘community school model sustainable’.
12
CAS 21st Century CLC Grant Evaluation 2010-2013
13
3. Findings
In this section we present key findings from the evaluation based on a selection of key
outcomes articulated in the Theory of Change.
3.1 Youth Outcomes
Academic Achievement
One of the key youth outcomes articulated in the Theory of Change is ‘Young people succeed
academically’. As illustrated in Chart 5, proficiency in Math between 2010 and 2012 has
increased each year for both schools and for after school students within each school. In
particular:
At Salomé Ureña Campus, average % proficiency increased from 39% in 2009-10, to 45%
in 2010-11 and 55% in 2011-12.
At Mirabal Sisters Campus, average % proficiency increased from 52% in 2009-10 to 58%
in 2010-11 and 59% in 2011-12.
Over this period math proficiency of after school students was 46% in 2009-2011 and
increased to 57% in 2011-12.
The new proficiency test aligned with the common core curriculum resulted in a 30.4 percent
decline in average scores for math in NYC in the 2012-2013 school year as compared to the
previous year2. The two schools under review also posted lower proficiency rates under the
new testing. These results are discontinuous from previous years, owing to the difference in
both curriculum and test content, and cannot be compared. Test results for 2013 will provide a
new baseline.
2 For discussion on the implications of the move to the new proficiency testing see the report 2013 New York State Common Core Test Results: New York City Grades 3-8. http://www.nyc.gov/html/om/pdf/2013/2013_math_ela_deck.pdf
Chart 6 below illustrates average proficiency in English Language Arts. Proficiency in ELA for
each school and in after school students has been lower and somewhat more variable than
proficiency in math between 2010 and 2013. For example:
At Salomé Ureña Campus the average proficiency in ELA was 28% in 2009-10, decreasing
to 23% in 2010-11 and then increasing to 32% in 2011-12.
At Mirabal Sisters Campus the average proficiency was 20% in 2009-10, increasing to
21% in 2010-11 and then 25% in 2011-12.
CAS after school students average proficiency increased from 23% in 2009-10, to 24% in
2010-11 and 28% in 2011-12.
Chart 6: Average % Proficiency in ELA
0%
10%
20%
30%
40%
50%
60%
70%
2009-10 2010-11 2011-12
Average % Proficient inMath Salome Urena DeHenrique(IS218,MS322,CCAA293)
Average % Proficient inMath Mirabal SitersCampus (MS319,MS324)
Average % Proficient inMath CAS AfterschoolStudents
0%
5%
10%
15%
20%
25%
30%
35%
2009-10 2010-11 2011-12
Average % Proficient inELA Salome Urena DeHenrique(IS218,MS322,CCAA293)
Average % Proficient inELA Mirabal SitersCampus (MS319,MS324)
Average % Proficient inELA CAS AfterschoolStudents
CAS 21st Century CLC Grant Evaluation 2010-2013
15
The relatively low levels of proficiency in ELA, compared to scores in math, is correlated with
the high numbers of English Language Learners (ELL) in the schools. This can be seen if the
figures for the constituent schools are disaggregated. For example, in Salomé Ureña over 45
percent of the student body at I.S. 218 in 2011-2012 were English Language Learners and the
proficiency level in ELA in this period was 18 percent. By contrast, M.S. 293, another
constituent school of Salomé, had an enrollment of only 7.3 percent of ELL students and, at 56
percent, showed much greater proficiency in ELA.
Attendance
A key precondition in the Theory of Change for students to succeed academically is having ‘High
Attendance’ in school. Charts 7 and 8 below illustrate average attendance rates for each
community school and their constituent schools from 2010 to 2013 compared to an
approximate city average.3
As illustrated there is some variation in attendance rates among the constituent schools at both
Salomé and Mirabal. From 2010 to 2013 CCAA 293 had an attendance rate above the city
average while I.S. 218 was somewhat lower. At Mirabal Sisters Campus, both schools had an
average attendance rate consistently above the city average throughout this period.
Chart 7: Attendance Rate at Salomé Ureña Campus
Chart 8: Attendance Rate at Mirabal Sisters Campus
3 The city average was calculated by the evaluators using the New York City Department of Education Progress
Reports.
89.9% 90.3% 91.2%
89.2%
91.8% 91.1%
92.3% 91.6%
95.0% 94.3%
95.2% 94.8%
86.0%
88.0%
90.0%
92.0%
94.0%
96.0%
2009-10 2010-11 2011-12 2012-13
IS 218
MS 322
CCAA 293
94.2%
93.4%
94.3% 94.4%
95.0% 95.0% 94.9% 94.8%
92.5%
93.0%
93.5%
94.0%
94.5%
95.0%
95.5%
2009-10 2010-11 2011-12 2012-13
MS 319
MS 324
CAS 21st Century CLC Grant Evaluation 2010-2013
16
Chronic/Severely Chronic Absenteeism
Even a school with high attendance rates can have high ‘chronic or ‘severely chronic’ absentee
rates – for example, because different students take turns being absent on a daily basis. The
attendance rate might be 95 percent but when the absences are added together, they can
accumulate and student(s) can miss a month or more of school over the course of the school
year. In New York City, a student is ‘chronically’ absent if he/she misses 20-37 days of school in
a year and ‘severely chronically’ absent if he/she misses 38 days or more of school. 4
The Children’s Aid Society has been taking leadership on analyzing and dealing with chronic
absenteeism in its community schools. Figures on chronic absenteeism in both schools show
that this leadership is having an impact. As outlined in Chart 9 below, ‘chronic’ and ‘severely
chronic’ absentee rates declined for both community schools between 2010-11 and 2011-12.
The ‘chronic absentee’ rate in Salomé Ureña Campus decreased from 16 percent to 15 percent
and the ‘severely chronic’ absentee rate decreased from six percent to five percent. Similarly in
Mirabal Sisters Campus, the ‘chronic absentee’ rate decreased from 11 percent to 8 percent but
the ‘severely chronic absentee’ rates was maintained at four percent. These percentage
differences appear low, but the absolute numbers of chronically absent students are relatively
small, so a small percentage difference means a few individuals have been turned around.
Chart 9: Chronic and Severely Chronic Absentee Rate Comparison between Salomé Ureña and
Mirabal Sisters Campus (2010-11 and 2011-12)
4 For fuller analysis of chronic absenteeism see for example the resources section of the National Center for Community Schools and reports for example, the National Center for Children in Poverty Report Present, Engaged, and Accounted For (Chang at el, 2008).
‘Chronic’ and ‘severely chronic’ absentee rates for Salomé Ureña and Mirabal Sisters Campus
were lower for New York City as a whole in 2011-2012. This is illustrated in Chart 10 below
which shows that Mirabal Sisters Campus and Salomé Ureña Campus had ‘chronic absentee’
rates of eight and 15 percent respectively compared to 19 percent for the city as a whole. The
two schools also had ‘severely chronic absentee’ rates of four and five percent respectively
compared to six percent for the city.5
Chart 10: 2011-12 Chronic and Severely Chronic Absentee Rate: Salomé Ureña and Mirabal
Sisters Campus Compared to Citywide Average
Key personnel in both schools highlighted the capacity of the community school model to
engage with all stakeholders, particularly with parents, as being critically important to
addressing absenteeism and chronic absenteeism issues. For example, travel between the
United States and the country of origin of students from immigrant communities (who
comprise a significant proportion of the student body in both schools) was found to be one
factor in absentee rates for some students. The success of the community school in engaging
with parents provided effective approaches to facilitate such travel in ways that reduced
student absenteeism.
5 Data derived from Balfanz and Byrnes (2008). Meeting the Challenge of Combating the Chronic Absenteeism: Impact of the NYC Mayor’s Interagency Task Force on Chronic Absenteeism and School Attendance and Its Implications for Other Cities. http://www.nyc.gov/html/truancy/downloads/pdf/meeting_the_challenge_of_combating_chronic_absenteeism.pdf