FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 1 Gaylen Moore Program Evaluation Services 304 West 89 Street, New York, New York 10024 Telephone / Fax (212) 724-8812 gmoore1@nyc.rr.com Evaluation of the Wildlife Conservation Society, The New York Botanical Garden, Lehman College Department of Early Childhood and Elementary Education Fund for Improvement of Secondary Education (FIPSE) Outside-the-Box Professional Development Program Year 3 2013-14 Submitted by Gaylen Moore
40
Embed
Gaylen Moore Program Evaluation Servicesfscdn.wcs.org/2015/10/05/6oihteahy5_FIPSE_3yr_Report_11...FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 4 SCOPE
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 1
Gaylen Moore Program Evaluation Services 304 West 89 Street, New York, New York 10024
Wildlife Conservation Society, The New York Botanical Garden,
Lehman College Department of Early Childhood and Elementary Education Fund for Improvement of Secondary Education (FIPSE)
Outside-the-Box Professional Development Program
EVALUATION REPORT Year 3
2013-14
I. INTRODUCTION PROGRAM DESCRIPTION The Wildlife Conservation Society/Bronx Zoo (WCS) concluded the third year of the grant awarded by the Fund for Improvement of Secondary Education (FIPSE) in June 2013. The Outside-the-Box (OTB) program was designed to introduce science to pre-service elementary teachers in the Lehman College Department of Early Childhood and Elementary Education and model the use of outdoor settings to teach inquiry science and integrate science into all subject areas. Project staff educators from the Wildlife Conservation Society/Bronx Zoo (WCS) and The New York Botanical Garden (NYBG) delivered one full-credit science methods course called Learning and Teaching Science in Childhood Settings in June 2013 to conclude the third year of implementation. The course was delivered at both the New York Botanical Garden and WCS sites. To help students complete their assignments and utilize the resources of these institutions to facilitate planning outdoor learning activities, the Outside-the-Box program provided semester-long free passes for students to visit both sites and an online website which provided access to a rich variety of content and instructional resources. II. EVALUATION METHODOLOGY EVALUATION GOALS AND OBJECTIVES Program Goals
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 3
The FIPSE Outside-the-Box program was designed to answer the following questions: 1) What science attitudes and disciplinary understandings do participants hold before beginning the program? 2) What changes in science attitudes and disciplinary understandings do participants demonstrate by the end of their experiences? 3) What teaching and learning practices do Lehman teachers engage in to support participants’ science learning after conclusion of the
program? 4) How do participants’ new practices influence their students’ engagement and understanding of science? 5) How can the project most effectively disseminate the lessons learned so that others can replicate all or parts of the model?
Program Objectives Between Year 1 and Year 2, project staff revisited the goals for achieving program effectiveness that were established in Year 1, and determined that not all the proposed objectives were being addressed by program activities. Objectives 3 and 4 cold not be addressed by the program or the evaluator due to a lack of response from participants after they graduated from Lehman. Consequently, the staff identified four areas in which program activities were aligned with program goals. The following goals are those that will be addressed in this report. It was expected that students completing the entire program of Outside-the-Box courses would be able to:
• demonstrate positive attitudes and knowledge about life science • design science lessons that utilized outdoor settings to facilitate student learning • demonstrate knowledge and ability to integrate science into other subject areas • increase confidence, comfort, and preparation to apply the skills and knowledge acquired during the program
In Year 3, the evaluator conducted the following evaluation activities:
• Pre/Post Knowledge Assessment administered in the Learning and Teaching Science in Childhood Settings course to assess changes in science content knowledge.
• Pre/Post Participant Survey to assess interest and attitudes toward science, the integration of science into other subject areas, and use of outdoor settings to teach science.
• Observations of Outside-the-Box science course sessions to assess quality of project implementation • Course Evaluation Survey to assess participants’ perceptions of the quality and impact of project implementation
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 4
SCOPE OF REPORT This report includes an analysis of the implementation of the Teaching and Learning Science in Childhood Settings science course, which concluded the grant in June 2013. The analysis includes: 1) an analysis of Year 3 implementation; 2) a three-year cohort comparison of outcomes; and 3) an aggregated analysis of outcomes for all three cohorts combined. The evaluator was unable to answer research questions 3 and 4 to assess how participants utilized content and practices learned in the Out-of-the-Box program in their own classrooms. Repeated efforts to schedule focus groups, interviews, and visits to participants’ classrooms after they graduated from Lehman College resulted in no responses. Another factor that limited collection of data to answer these questions was that not all Lehman Out-of-the-Box graduates obtained teaching positions when they graduated. III. FINDINGS YEAR 3 ANALYSIS IMPLEMENTATION The Learning and Teaching Science in Childhood Settings course was taught at the New York Botanical Garden and the Bronx Zoo in nine four-hour classes in June 2013, five sessions at the Garden and four sessions at the Zoo. The curriculum addressed the nature of science (making observations, asking questions, collecting and interpreting data) and highlighted basic concepts of life science (defining living things, identifying biotic and abiotic elements – and their interactions -- in an ecosystem, and exploring biodiversity). The course activities included: exploring habitats, adaptations and survival strategies; identifying relationships and changes in the ecosystem; recognizing the diversity of life; and examining human impact on the environment. The course also touched on components of physical, planetary, and earth science. During this course, students conducted their own plant growth experiments; visited exhibits at NYBG and the Bronx Zoo; engaged in scientific investigations and interactive activities outdoors at these sites that were designed to facilitate their understanding of science, and provide opportunities for them to practice the skills and processes of scientific inquiry. A total of 15 students enrolled in the Lehman College Department of Early Childhood and Elementary Education participated in the final year of the Out-of-the-Box grant activities. QUALITY OF IMPLEMENTATION
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 5
In Year 3, the Outside-the-Box program continued to demonstrate overall high quality implementation, incorporating the evaluator’s recommendations from previous years to improve the course. The course modifications reflected a more simplified and streamlined approach to the plant experiment as the focus of the course, based on the course time constraints and the previous assessment of participants’ limited prior knowledge related to controlled experiments. Consequently, participants’ acquired greater understanding of the scientific skills and processes involved in a controlled experiment, and were able to build their knowledge related to plant growth and apply their skills to the process of science investigation in general. Participants’ Perceptions of Implementation: Year 3 At the conclusion of the course in June 2013, the evaluator administered a course evaluation survey with a five-point scale (1=Very Low to 5= Very High) to assess participants’ perceptions of the quality of implementation. Findings indicate that participants found the Learning and Teaching Science in Childhood Settings course to be effective in achieving the program’s objectives to improve participants’ capacity to:
• demonstrate positive attitudes and knowledge about life science • design science lessons that utilized outdoor settings to facilitate student learning • demonstrate knowledge and ability to integrate science into other subject areas • increase confidence, comfort, and preparation to apply the skills and knowledge acquired during the program
Participants rated the science course highly overall by participants for the quality of implementation; 96 percent gave positive ratings to their overall engagement in the course, and 86 percent rated the quality of the course as “High” to “Very High.”
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 6
Science Course Evaluation, Year 3 N= 14
93% 79%
100% 100% 93% 93% 86%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Your overall self-engagement in
the module
Your knowledge of the module
content
Your increased knowledge of wildlife/animal
conservation and WCS
Your increased knowledge of plants and the
natural environment and
NYBG
Your increased awareness of
outdoor sites to support teaching
and learning
Overall likelihood you will use or
apply information obtained from this course in
your classroom
Overall quality of course
experience
% ra
ted
high
/ver
y hi
gh
Learning and Teaching Science in Childhood Settings Science Course Evaluation
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 7
• 96 to 100 percent of participants reported the science course contributed to increasing: knowledge about wildlife and animal conservation and plants and the natural environment; awareness of outdoor sites to support teaching and learning; and the likelihood that they would use what they learned in the course in their future classrooms.
• The majority of participants reported the amount of information and the range of topics covered in the course were “just right.” They also believed the assumed level of prior knowledge and the amount of time that instructors devoted to each topic was just right. Fewer participants reported that the overall length of the science course was just right. Those who were dissatisfied with the length of the course felt it was too long. Some felt the nine sessions were too much.
79% 86% 79% 86% 57%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Assumed level of your prior knowledge
Amount of information presented
Amount of time devoted to each topic
or learning experience
Range of topics Overall length of module
% ra
ted
just
righ
t
General Impressions of Science Course Year 3
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 8
• All participants were satisfied that the science course introduced them to new and innovative ideas (100 percent) and prepared them to teach science in other subject areas using the facilities and recourses of WCS and NYBG (100 percent). Almost all participants (93 percent) reported that the science knowledge they gained enhanced their ability to teach science and integrate it into other subject areas. Fewer participants were satisfied with their ability to implement the pedagogical practices modeled in the course to teach science.
100% 93% 79%
93% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
I was introduced to new or innovative
ideas
The science knowledge gained from this module enhanced
my ability to teach science to elementary
children
The pedagogical practices and
instructional strategies modeled in this module enhanced my ability to
teach science effectively to
elementary children
The module facilitated my ability to integrate wildlife and the natural environment into my instruction in other
subject areas
The module prepared me to teach science in
other subject areas using the Wildlife
Conservation Society and the New York Botanical Garden
% ra
ted
agre
e/st
rong
ly a
gree
Impact of Science Course
Year 3
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 9
• Participants believed the WCS and NYBG websites were clear and easy to navigate for the purposes of the course assignments and submissions. Further, they reported the online resources would be relevant to their future role as elementary teachers, and would facilitate their ability to incorporate wildlife and the natural environment into lesson planning and instruction.
86% 93% 93% 93%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
The WCS and NYBG websites were clear and easy to navigate for my
purposes
The resources were easy to access
The resources were relevant to my role as a
future elementary teachers
The resources facilitated my ability to incorporate wildlife and the natural
environment into lesson planning and instruction.
% ra
ted
agre
e/st
rong
ly a
gree
Rating of Online Component Year 3
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 10
• Participants found the interactive activities to be the most effective strategy in facilitating their learning, followed by the discussions, then pictures and other visuals used by instructors to teach the course.
86%
57%
100%
71% 93%
79%
50%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Pictures and other visuals
Lecture presentation
Interactive Activities
Suggested resources
Discussions Assignments Readings
% ra
ted
influ
entia
l/mos
t inf
luen
tial
Contributions of Program Components to Learning in the Science Course Year 3
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 11
An analysis of participants’ use of NYBG and WCS free passes provided by these institutions to do their assignments for the course shows that a majority of participants used their passes to visit the sites.
• On average, participants visited the Bronx Zoo two times and the New York Botanical Garden once • Participants visited both institutions to prepare coursework, and plan lessons or field trips to meet the course requirements.
Participants Open-Ended Responses to Science Course In response to questions about how they planned to use what they learned in the science course in their own classrooms, when they became teachers, participants cited the following activities and practices:
• The adaptation game with the candy. • The lima bean experiment. • I will use the outdoor resources to teach all subjects. • I would teach students about biotic, abiotic, ecosystems and the environment in creative ways, such as taking them on a leaf
identification field trip. • Using outdoor resources to further ELA, math lessons with science.
79% 71%
0%
20%
40%
60%
80%
100%
% u
sed
Use of passes to visit zoo or garden Year 3
Bronx Zoo Passes New York Botanical Garden Passes
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 12
• I learned being in this course that there are many different ways to teach science. Science can be done in the outdoors, which makes students more excited to learn.
• The use of follow-up sessions as a means for differentiation to make sure students understand objectives. When asked how they would integrate science into other disciplines, participants reported they would adopt the following activities or practices:
• Nature walk counting living and non-living thing. • Incorporate ELA & math in my outdoor lesson plans such as measuring trees and writing a leaf story. • Graphing ,science research, math, reading, science literature-ELA. • When I visited the classroom at the Bronx Zoo where a lesson was being conducted I saw how science could be incorporated with
literacy. For example, the teacher had the students write poems about the animal they were observing. • Students can take their observations and claims to research more background on a topic and unite informational text on the topic. • Use of Venn diagrams for comparing and contrasting ,then students can write a passage on observation from the diagram,
incorporating literacy. When asked how they would use NYBG and WCS with their students, participants reported:
• Use the Bronx zoo to study animals & habitats. • I plan to focus on specific exhibits and teach about them prior to visiting. • Children can research a specific animal then take a field trip to the zoo and observe the animal they researched. • I saw how the Bronx Zoo and NYBG can be used to teach many different lessons about plants and animals.
When asked to cite the greatest benefit of the course, participants reported:
• It was helpful in using outdoor environments. • Overall my experience with the course was pleasant. • The course was very informative and I found it to be engaging. In a 4+ hour class, it was good to be able to move around and explore
while learning. • This program made science less frightening and fun. • This program strengthened my knowledge about science and I was surprised at my knowledge from the pre test to the post test. • Very exciting, interesting and informational class. Enjoyed the class since I was about to learn new things on how I can teach science
in my future class. • I loved this class. It wasn't what I expected. I now aspire to be that awesome science teacher I never had. I am soooo excited and
ready to learn more. • This program models great and useful methods of using outdoor spaces in all school subjects.
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 13
IMPACT ON PARTICIPANTS: YEAR 3 To assess the impact of the Outside-the-Box program on the Lehman College pre-service elementary education students, and the program’s effectiveness in achieving its revised objectives, the evaluator used two instruments that were used in the evaluation of each of the previous years of the program. The instruments included: 1); a pre/post knowledge assessment administered to students in the Learning and Teaching Science in Childhood Settings science course, and 2) a pre/post attitude survey administered to students before and after participation in the science course to assess their personal beliefs and attitudes toward science, knowledge of life science, attitudes toward inquiry practice, and attitudes about using inquiry in outdoor and out-of-the classroom settings to facilitate learning. Impact on Participant Knowledge: Year 3 The pre-post knowledge assessment was administered to participants before and after their participation in the Learning and Teaching Science in Childhood Settings science course. The instrument had 22 items, representing a combination of multiple choice and open-ended questions related to key concepts of life science. Findings indicate that the program had a positive impact on increasing participants’ knowledge of the content covered in the course.
• Participants showed tremendous gains in their knowledge assessments from pre to post test. On average participants had a percentage gain of 20 percent. A paired t-test showed that the gains were significant.
62.93% 82.00%
0.00%
50.00%
100.00%
% c
orre
ct
Learning and Teaching Science in Childhood Settings Science Course
Pre/Post Knowledge Assessment Year 3
Pre-Know N=15 Post Know N=14
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 14
Impact On Participants’ Attitudes: Year 3 Analysis of survey data collected in the science course indicate the program achieved its objective to promote a positive impact on participants’ attitudes about using outdoor settings to facilitate student learning in science and other disciplines.
• By the end of the Year 3 science course all participants believed science was important for all children to know. The course also developed more positive attitudes and beliefs among participants about their personal interest in science and confidence in understanding of science. The experience did not result in a greater interest in learning more science, however.
60%
20%
73%
33%
80% 69%
15%
69% 77%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
I have a personal interest in science
I read science's books and articles for pleasure and
learning
Interest in learning more science
Confident in my understanding of
science
The importance for children to know
science
% ra
ted
agre
e/ag
ree
to a
gre
at e
xten
t
Science Course Participants' Personal Attitudes/Beliefs: Year 3
Pre N=15
Post N=13
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 15
• Participants overall showed great gains in their confidence about their content knowledge concerning these key concepts of life science and topics about the environment. The gains were significant.
Science Course Participants' Confidence in Content Knowledge: Year 3
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 16
• Participants overall showed gains in their beliefs about their ability to use what they learned in the course to facilitate student learning. The program appeared to be most effective in teaching participants how to use the NYBG and Bronx Zoo to generate interest and excitement about science.
Use essential
questions to structure science lessons
Use observations
of specimens
or phenomena to stimulate
students
Provide reading
opportunities to develop students'
understanding of science
integrate science into
math, literacy, ss, and other disciplines
use questioning strategies to
elicit students
ideas
use collaborative small group
activities
use the NY botanical
garden and bronx zoo to
generate interest and excitement
about science
engage students in
observations or hands-on investigation
s in the classroom
use outdoor sites, parks, museums, and sites
outside the classroom to
conduct observations
or investigation
s Pre N=15 73% 73% 87% 73% 80% 87% 67% 87% 93% Post N=13 92% 92% 85% 100% 92% 100% 100% 100% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
% ra
ted
agre
e/st
rong
ly a
gree
Science Course Pedagogy: Year 3
To facilitate students' science learning it's important to:
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 17
• Participants significantly increased confidence on all parameters measured in their ability to teach science, achieving the program’s objectives in this sphere. They reported increased confidence in their ability to identify science inquiry, design and teach developmentally appropriate lessons using inquiry; address the science content standards; integrate the principles of environmental education; integrate science teaching and learning into science learning and other subject areas; and use outdoor settings to facilitate student learning.
THREE-YEAR COHORT COMPARISON Three-Year Knowledge Comparisoon In this final year of the project, the evaluator compared findings for the three years of the Learning and Teaching Science in Childhood Settings course to assess the program’s long-term effectiveness. The evaluator examined the pre/post knowledge and pre/post attitudes of each cohort participating in the science course in May/June 2011, 2102, and 2013.
identify the elements of an effective
science inquiry lesson
designing a developmentally appropriate
science inquiry lesson
designing a developmentally appropriate
science inquiry in
connection with nature
teach a developmentally appropriate
inquiry science lesson
address science content
standards in inquiry lesson
incorporating the principles
of environmental education into inquiry lesson
intergrate science
teaching and learning into other subject
areas
use outdoor settings to facilitate students' science learning
Pre N=15 33% 47% 53% 40% 40% 47% 47% 67% Post N=13 92% 77% 92% 77% 92% 92% 100% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
% ra
ted
agre
e/ag
ree
to a
gre
at e
xten
t Science Inquiry Practice: Year 3
Participants' confidence in ability to:
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 18
Findings show that the program was increasingly effective in achieving its objective to build participants’ knowledge of science concepts related to plant and animal life.
*only post scores shown
• A paired-sampled t-test was conducted to assess the significance of the outcomes of the pre/post knowledge assessment. There was a
significant difference in the scores for the pre and post-knowledge assessment across all Cohorts (p<. 000). Cohort 3 demonstrated the highest average scores of any cohort. This finding provides further evidence of the program’s effectiveness in achieving its goal to increase participants’ knowledge of life science.
Three-Year Attitude Comparison To assess the program’s effectiveness over the three years in developing more positive attitudes among participants about learning science and teaching science, the evaluator examined pre/post attitude survey findings from participants in all three cohorts of the science course in June
Science Course Knowledge Domain: Cohort Comparison of Knowledge Assessment Scores
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 19
2011, 2012, 2013. On a Likert scale of 1 (Strongly Disagree) to 5 (Strongly Agree), participants rated their attitudes about science content knowledge, teaching science, and implementing science. The attitude findings have been organized into three domains: 1) the Knowledge Domain which addresses the science content knowledge objective; 2) the Teaching Domain, which addresses the objectives in using outdoor settings and integrating science into other subject areas, and 3) the Application Domain, which addresses the objective related to students’ confidence and comfort in using newly acquired knowledge and skills in their future classrooms. Analysis of the data in all three domains combined shows that participants reported a total positive increase in attitudes. The increase in positive attitudes across the three cohorts of participants was statistically significant (p < .05). These findings indicate the program achieved its objectives to develop and promote more positive attitudes about participants’ interest in science and confidence in their ability to incorporate knowledge, strategies, and practices they learned in the course when they got their own classrooms. Knowledge Domain: Three-Year Cohort Comparison
Science Course Knowledge Domain: Cohort Comparison of Attitudes About Science,
by Question
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 20
• There was a significant difference in scores for the overall pre attitude beliefs and post attitude beliefs for all cohorts individually and aggregated (p<. 05). These findings provide evidence that the science course contributed to achieving the program’s objective to increase participants’ interest and attitudes about increasing science knowledge for themselves and their students.
*only post scores shown
• When assessing participants confidence in science knowledge, findings show there was a significant difference in participants’ pre and
post confidence in their knowledge of key concepts of life science for Cohorts 2 and 3 (p<.05) after participating in the science course. This finding provides evidence of the increasing effectiveness of the WCS and NYBG educators in teaching these concepts. The
My current knowledge of life science is adequate for elementary students
ConLident about classify objects as living or non-‐
living
ConLident about describing the
role of structure and function within plants and animals
ConLident about recognizing that traits of
living things are inherited, acquired or learned
ConLident explaining how the adaptations of species depends on
their environment
ConLident identifying life cycles and processes of growth and reproduction
ConLident knowing about the abiotic environment
ConLident about relationships between biotic and abiotic
components of the ecosystem
ConLident knowing how humans have changed
environment, positively & negatively and its effects
Science Course Knowledge Domain: Cohort Comparison of Con>idence in Science Knowledge
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 21
program’s effectiveness in contributing to this outcome grew each year, as is evident in the smaller increases in Cohort 1 findings compared to Cohort 2 and 3 findings.
Teaching Domain: Three-Year Cohort Comparison
*only post scores shown
• There was a significant difference across all participants in pre and post confidence in their ability to teach life science across all cohorts
(p<.000) after participating in the Out-of-the-Box science course.
The importance of integrating science into math, literacy, social studies and other disciplines
The importance of using New York
Botanical Garden and Bronx Zoo
The importance of using outdoor sites, parks, museums
ConLident in using outdoor settings to facilitate students' science learning
ConLident in integrating science teaching and learning into other subject
Science Course Teaching Domain: Cohort Comparison of A7tudes About Science Teaching
Cohort 1
Cohort 2
Cohort 3
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 22
Attitude Domain: Three-Year Cohort Comparison
*only post scores shown
• A comparison of the pre/post outcomes for all application items combined indicates that there was a significant difference in Cohorts 2
and 3 pre and post confidence about using the inquiry practices they had learned as participants in the science course (p<.05). Cohort 1 did not show any significant differences. These results demonstrate that both cohorts 2 and 3 improved their confidence in using inquiry in and out of the classroom.
ConLident in identify the elements of an effective science inquiry lesson
ConLident in designing a
developmentally appropriate
science inquiry lesson
ConLident in designing a
developmentally appropriate
science inquiry in connection with
nature
ConLident in teach/facilitate a developmentally appropriate
inquiry science lesson
ConLident in addressing
science content standards in inquiry lesson
ConLident in incorporating the principles of environmental education into inquiry lesson
At the conclusion of the Teaching and Learning Science in Childhood Settings course each year, participants were asked to rate the quality of implementation and the impact of the experience on their knowledge and attitudes about outdoor learning. The extent to which participants reported they were engaged overall in the course was rated highly across all three cohorts.
• In general, the findings show that the program achieved its goals. Participants in Cohorts 1 and 2 believed they increased their knowledge of course content (91 and 93 percent), while Cohort 3 did not rate their increase in knowledge as highly as the previous cohorts (79 percent). All cohorts believed they increased their knowledge of the course content of wildlife/animal conservation and the Bronx Zoo. Cohorts 2 and 3 rated the science course very highly for increasing their knowledge of plants and the natural environment and NYBG. Participants overwhelmingly stated the course increased their awareness of outdoor sites to support teaching and learning,
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 24
as well as their aspirations to apply this knowledge when they had their own classrooms. Finally, Cohorts 1 and 2 rated the overall quality of the course as “excellent”, while Cohort 3 believed the course was “very good”.
• Participants’ general impressions about the science course were rated “just right” for all three cohorts, when it came to assumed level of knowledge and the amount of information presented during the course, the time devoted to wildlife and natural environment topics. There were mixed feelings across cohorts when it came to rating the overall length of course. Cohort 1, generally believed the course length was “just right”, while Cohort 2 deemed they would like to see the course be extended to a full-semester course, rather than this condensed version. Finally Cohort 3, believed the course was too long.
Assumed level of your prior knowledge
Amount of information presented
Amount of time devoted to each topic or learning experience
Cohort Comparison: General Impressions of Science Course
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 25
• The findings show that the high quality implementation of the science course contributed significantly to achieving the program’s key objectives. Participants across all three cohorts overwhelmingly agreed the science course increased their knowledge of life science and their ability to teach it to elementary students. There were mixed reactions to the course’s facilitation of the ability to integrate life science into other subjects in Cohorts 1 and 3, but Cohort 1 strongly believed the course increased their pedagogical practices and instructional strategies “to a great extent”. Cohort 3 believed this statement to a moderate extent. Both Cohorts 2 and 3 believed the course facilitated participant’s ability to integrate wildlife and the natural environment into their instruction in other subject areas. All cohorts believed the course prepared them to use outdoor settings at WCS and NYBG to do it.
* NOTE: Due to changes in the course evaluation instrument, data was not collected for some items.
I was introduced to new or innovative
ideas
The science knowledge gained from this
course enhanced my ability to teach science to elementary children
The pedagogical practices and
instructional strategies modeled in this course enhanced my ability to
teach science effectively to
elementary children
The course facilitated my ability to integrate
wildlife and the natural environment into my instruction in other subject areas
The course prepared me to teach science in other subject areas using the Wildlife
Conservation Society and the New York Botanical Garden
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 26
• The online components of the science course were generally rated highly across Cohorts 2 and 3 because the websites were easy to access and navigate. Participants found the resources to be relevant to their future work as teachers, and contributed to their learning through the organization of the materials. The online components included engaging content; facilitated opportunities to share thoughts and ideas; and clearly defined the course assignments which were posted and submitted online. Cohort 1 did not believe the WCS and NYBG websites were as easy to navigate or access, but did believe the resources facilitated their ability to incorporate information into their own lesson planning. This difference may be due in part to improvements made to both the WCS and NYBG websites over the period of this grant, and the adoption of a new platform (moodle) in Year 2.
The WCS and NYBG websites were clear and easy to navigate for my purposes
The resources were easy to access
The resources were relevant to my role as a future elementary teachers
The resources facilitated my ability to incorporate wildlife and the natural environment into lesson planning and
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 27
• In rating the pedagogical elements of the course that contributed most to their learning, Cohort 2 and 3 participants showed consensus in their ratings of the interactive activities and discussions as having the most positive impact on their learning. Cohort 2 further believed pictures and other visuals, lecture presentation, online resources and readings were also factors contributing to their learning, but slightly less so. Cohort 3, in comparison only believed pictures and visuals, assignments, and suggested resources were influential. Cohort 1 was most critical, rating only the interactive activities as influential, while rating all other components as ‘not helpful ‘or ‘useless.’ This finding suggests the program improved the quality of the other instructional strategies in Years 2 and 3. Both Cohorts 1 and 3 believed the reading activities had the least influence on their learning in the science course.
Cohort Comparison: Contributions of Components to Learning in the Science Course
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 28
AGGREGATED THREE-YEAR ANALYSIS To assess the performance of the program overall at the end of the grant period, the evaluator prepared an aggregated analysis of the three-year comparison data that were discussed above. The findings for the three-year aggregated analysis are presented below.
N=62
• A paired-sampled t-test was conducted to compare pre and post knowledge assessment scores for the Learning and Teaching Science in Early Childhood Settings course across all cohorts. There was a significant difference in the scores for the pre knowledge assessment (M=59.19) and post-knowledge assessment (M=75.77). The two-tailed P value is less than 0.0001. By conventional criteria, this difference is considered to be extremely statistically significant. This finding suggests the program overall was effective in achieving its objective to increase participants’ knowledge of important life science concepts.
59% 76%
0%
20%
40%
60%
80%
100%
Learning and Teaching Science in Childhood Settings Science Course
Pre/Post Knowledge Assessment: Aggregated
Pre_Know Post_Know
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 29
• Participants’ attitudes about the value of science education changed over the course of the three-year grant period. By the end of the course nearly all participants believed science was important for all children to know. Participant’s attitudes about reading science books for pleasure, interest in learning more about science, and their confidence in their understanding of science nearly doubled. Participant’s personal interest in science increased slightly, but not as dramatically as the other attitudes and beliefs measured.
50%
19%
48% 54% 60% 59%
31%
76% 90% 84%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
I have a personal interest in science
I read science's books and articles for pleasure and
learning
Interest in learning more science
ConLident in my understanding of
science
The importance for children to know
science
Science Course Participants' Personal Attitudes/Beliefs: Aggregated
Pre (n=48) Post (n=49)
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 30
• Participants overall showed extremely significant (p<.0001) gains in confidence in their content knowledge concerning life science and the environment.
My current knowledge of life science is adequate for elementary
Science Course Participants' Confidence in Content Knowledge: Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 31
• Participants overall showed extremely significant (p<.00) gains in their beliefs about facilitating student learning. Using the NYBG and Bronx Zoo to generate interest and excitement about science showed the greatest gains. This outcome was a key objective of the program, and achieving it was a key accomplishment.
To facilitate students' science learning it's important to: Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 32
• Participants demonstrated extremely significant (p<.000) increases in their confidence and ability to identify science inquiry, design and teach developmentally appropriate lessons, and integrate the principles of environmental education into science learning and other subject areas. The outcomes indicate that the program was effective in achieving the objectives to instill understanding of inquiry pedagogy and interdisciplinary learning.
identify the
elements of an
effective science inquiry lesson
designing a
developmentally
appropriate science
inquiry lesson
designing a
developmentally
appropriate science inquiry in connectio
n with nature
teach a developm
entally appropriate inquiry science lesson
address science content
standards in inquiry lesson
incorporating the
principles of
environmental
education into
inquiry lesson
intergrate science teaching
and learning
into other subject areas
use outdoor settings
to facilitate students' science learning
Pre (n=48) 58% 67% 63% 60% 65% 65% 58% 69% Post (n=49) 92% 86% 90% 88% 92% 84% 86% 90%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Science Inquiry Practice Participants' con>idence in ability to: Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 33
(N=59)
• In course evaluations administered at the end of each session, participants were highly positive about their experience in the course. On six of the seven items measured, 91 to 96 percent rated them positively, indicated they were very favorable (ratings of 4 or 5) about the overall quality of the course experience; overall engagement in the course; the impact of the course on increasing their knowledge of animal life and plants; and the ability to use outdoor resources to facilitate science learning in science and other subject areas. When asked if they would use their newly acquired learning when they had their own classrooms, 96 percent of participants reported they would. This findings supports the participants’ committment to applying the program’s knowledge and skills, even though no data was collected to provide evidence of participants’ actual use. of what they learned in the course.
92% 88% 95% 96% 96% 96% 91%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% rated high/very high
Science Course Evaluation: Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 34
• Findings from the aggregated course evaluation data show that the majority of participants were satisfied that the amount of information and the range of topics covered in the course, as well as the amount of time devoted to each topic over the three years were “just right”. They also believed the course was directed to a level of prior knowledge that was “just right.” Most participants believed the overall length of module was appropriate but more than a fourth of participants felt it could have been longer or shorter. Both opinions were expressed by participants in the course evaluation data.
80% 91% 86% 88% 71%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Assumed level of your prior knowledge
Amount of information presented
Amount of time devoted to each topic
or learning experience
Range of topics Overall length of course
% rated just right
General Impressions of Science Course: Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 35
• All cohorts of participants combined were unanimous in reporting that the science course introduced them to new and innovative ideas. Most reported that the course prepared them to teach science and to teach science in other subject areas using WCS and NYBG. These outcomes suggest that the program achieved another of its key objectives.
100% 95% 97%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
I was introduced to new or innovative ideas
The science knowledge gained from this module
enhanced my ability to teach science to elementary
children
The course prepared me to teach science in other subject
areas using the Wildlife Conservation Society and the New York Botanical Garden
% rated agree/strongly agree
Impact of Science Course: Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 36
• Three cohorts of participants believed the WCS and NYBG websites and online support component were clear and easy to navigate. Further, they deemed the online components to be relevant to their future role as elementary teachers because they provided lesson planning guidance and other resources that could be used in their future classrooms. They also believed the online resources and facilitated their ability to incorporate wildlife and the natural environment into lesson planning and instruction.
84% 88% 95% 95%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
The WCS and NYBG websites were clear and easy to navigate for my
purposes
The resources were easy to access
The resources were relevant to my role as a future elementary
teachers
The resources facilitated my ability to incorporate wildlife and the natural environment into lesson planning and instruction.
% rated agree/strongly agree
Ratings of Online Component: Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 37
• Participants found the interactive activities to be the most effective instructional practices of the course in instilling the desired learning,
followed by the class discussions and educators’ use of pictures and other visuals. DISSEMINATION AND REPLICATION To meet the program’s final objective to disseminate the accomplishments of the Out-of-the-Box program and identify the effective elements of the program for possible replication, project staff and educators made presentations at national conferences throughout the three-year grant period. Presentations were made by both WCS and NYBG program staff in 2011, 2012, and 2013 at conferences of the following organizations: the North American Conference for Environmental Education (NAAEE), American Public Gardens Association (APGA), Association of Zoos and Aquariums (AZA), and the National Science Teachers Association (NSTA). The key findings presented at each conference are summarized below. The major overall theme of the presentations was to provide evidence from the evaluation findings to disseminate the idea that informal science institutions can play a key role in developing inquiry science teaching at the elementary school level.
76% 61%
93%
68% 80%
70% 57%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Pictures and other visuals
Lecture presentation
Interactive Activities
Suggested resources
Discussions Assignments Readings
Ax% rated in>luential/m
ost in>luential
Contributions of Components to Learning in the Science Course; Aggregated
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 38
NSTA Presentation Evaluation results after the first year (May 2011 through May 2012), suggest that Informal Science Institutions (ISIs) can play a significant role in promoting science literacy through teacher development. A pre/post comparison of the elementary pre-service teachers’ personal attitudes about science, content knowledge, attitudes toward inquiry, and use of inquiry in/out of the classroom shows that they reported a positive increase in attitudes after the program’s coursework. The surveys demonstrate that students were committed to using their newly acquired knowledge and skills when in their own classrooms -- a testament to the positive impact of this program on introducing science and outdoor settings in the context of other disciplines. Students’ confidence in their ability to implement science inquiry in and out of the classroom improved on all parameters. AZA Presentation The Outside-the-Box program is a demonstration of one way in which a partnership between ISIs and higher education institutions can create a unique opportunity for the preparation of future elementary teachers, with an emphasis on science and nature education. Findings show that the semester-long science course made significant gains in pre-service teachers’ attitudes about science and science education and contributed to increasing their knowledge of life science. The experience also increased their confidence and ability to apply the knowledge and skills acquired in the program. As a result of participating in the Out-of-the-Box Program, Pre-Service Teachers:
• Were better able to use informal science institutions as valuable educational resources • Demonstrated enthusiasm about science and science teaching • Increased their appreciation for the role of wildlife and animals in the environment • Developed skills in pedagogy and science instructional strategies • Demonstrated confidence in teaching science • Were able to incorporate science into other disciplines • Were able to recognize inquiry in science lessons
AGPA Presentation Informal science institutions represent a tremendous resource for teacher education. Science-rich institutions can play a leading role in providing opportunities that support teachers’ professional development and offer experiences beyond what most traditional teacher preparation programs provide. However, few informal science institutions provide programs for pre-service teachers and the effectiveness of these programs has not been extensively documented or broadly disseminated. The Out-of-the-Box program demonstrated one way in which partnership among gardens, zoos and higher education institutions can be formed to support teacher education.
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 39
The Out-of-the-Box program model was designed to contribute to the preparation of pre-service elementary teachers in collaboration with their formal graduate studies. Conference participants will be able to replicate elements of the Out-side=the-Box program through partnerships involved APGA institutions and universities. Students: • demonstrate greater confidence in using informal science institutions as valuable educational resources
• demonstrate enthusiasm about science and science teaching • Increasd appreciation for the role of wildlife and animals in the environment • Develop skills in pedagogy and science instructional strategies • Demonstrate confidence in teaching science • Are able to incorporate science into other disciplines
IV. CONCLUSION CONCLUSION The FIPSE Outside-the-Box program completed its third year of implementation in June 2014 with 15 students enrolled in the Lehman College Early Childhood and Elementary Education Program. Year 3 concluded with the third iteration of the Learning and Teaching Science in Childhood Settings science course in May/June 2013. The course continued to be implemented at a high level, incorporating the evaluator’s recommendations from previous years. Project staff and educators from WCS and NYBG delivered a curriculum that targeted the program’s objectives to prepare participants to:
• increase knowledge and develop positive attitudes about life science; • use the New York Botanical Garden and Bronx Zoo, and other outdoor settings to teach science • integrate inquiry and use of outdoor settings to each science in any subject area • utilize inquiry practices (questioning, observing, exploring phenomena) to facilitate science learning in and out of the classroom
Project staff and educators’ decision to modify the science course in Year 2 to focus more intensely on both modeling and explicitly addressing the practice of inquiry in the context of outdoor education, using the WCS and NYBG sites, paid off in Year 3. Findings showed that the program achieved its objectives to increase participants’ knowledge of life science, develop more positive attitudes about the importance of science, develop a greater personal interest in science, build confidence in their life science knowledge and use of inquiry strategies to teach science and incorporate science with other subject areas in outdoors settings.
FIPSE Out of the Box Program 13-14 Gaylen Moore Program Evaluation Services 40
A three-year cohort analysis of the data demonstrated the program was increasingly more effective with each new cohort in achieving the program’s objectives. A further analysis of the data, aggregating outcomes from all three years of the program provide largely significant evidence that the program was effective in developing participants’ ability to teach science using inquiry practices in outdoor settings. The three-year aggregated findings show that the pre-service elementary teachers:
• demonstrated positive attitudes and knowledge about life science • designed science lessons that utilized outdoor settings to facilitate student learning • demonstrated knowledge and ability to integrate science into other subject areas • increased confidence, comfort, and preparation to apply the skills and knowledge acquired during the program
One of the project’s other accomplishments was building participants’ confidence and a positive commitment to applying their newly acquired content knowledge and practices when they became teachers. Participants reported they planned to integrate life science content into their instruction across subject areas, adopt the practices of inquiry, and use outdoor settings to teach science. The successful components of the Out-of-the-Box program and the program’s overall successes in introducing pre-service teachers to the value of outdoor learning were disseminated by project staff and educators in several national conferences throughout the grant period. Overall, the key contribution of this program to the field was providing evidence that informal science institutions can play a key role in developing inquiry science teaching content and practice at the elementary school level.