Inspire Summer School, Salzburg, 18-20 August 2009
External evaluation of the INSPIRE project:
Innovative Science Pedagogy in Research and Education
By Yves Beernaert & Magda Kirsch, Educonsult
Comenius Transnational projectINSPIRE
Contract number: 2007-3430 / 001-001Project number – 133867-LLP-1-2007-1-BE-COMENIUS-CMP
Inspire Summer School, Salzburg, 18-20 August 2009
Why the INSPIRE project ?
• It supports the Lisbon agenda 2000– 2010: EU is to become the most competitive and dynamic knowledge-based economy in the world,
capable of sustainable economic growth with more and better jobs and greater social cohesion
• It contributes to the benchmarks 2010– Halve the rate of early school leavers, in order to achieve an EU-average rate of
10%or less.– Halve the level of gender imbalance among graduates in MST & secure an
significant increase of the total number of MST graduates,– Average % of 25-64 yrs in the EU with at least upper sec. education reaches 80%or
more– Halve % of low-achieving 15 year olds in reading, mathematical and scientific
literacy– Increase EU-average level of participation in LLL to at least 15% of working adult
Inspire Summer School, Salzburg, 18-20 August 2009
Lisbon agenda: Detailed workprogramme:
3 strategic objectives
1. Improving the quality and effectiveness of education and training systems in Europe
2. Facilitating access of all to education and training systems
3. Opening Up education and training systems to the wider world
Inspire Summer School, Salzburg, 18-20 August 2009
Strategic objective 1: Improving the quality and effectiveness of education and training systems in
Europe
• 1.1. Improving education and training for teachers and trainers
• 1.2.Developing skills for the knowledge society
• 1.3. Ensuring access to ICT for everyone• 1.4. Increasing the recruitment to scientific
and technical studies
Inspire Summer School, Salzburg, 18-20 August 2009
Strategic objective 2: Facilitating access of all to education and training systems
• 2.1. Open learning environments• 2.2. Making learning more attractive• 2.3. Supporting active citizenship, equal
opportunities and social cohesion
Inspire Summer School, Salzburg, 18-20 August 2009
Strategic objective 3: Opening Up education and training systems to the wider world
• 3.1. Strengthening the links with working life and research, and society at large
• 3.2. Developing the spirit of enterprise• 3.3. Improving foreign language learning• 3.4. Increasing mobility and exchange• 3.5. Strengthening European co-operation
Inspire Summer School, Salzburg, 18-20 August 2009
External evaluation of INSPIRE: AIMS
To assess:• the quality & relevance of activities, processes implemented• the quality and relevance of the ‘educational ‘ or pedagogical’
outputs or the results• the quality and relevance of the dissemination / exploitation /
valorisation actions • the sustainability of the project activities • the impact on various stakeholdersTo identify key success and failure factors of the overall projectTo propose measures that will improve effective project delivery in
future partnerships
Inspire Summer School, Salzburg, 18-20 August 2009
Elements of evaluation
• Relevance: contribution to specific needs at local, nat. level versus objectives of Comenius
• Efficiency: relationship input versus output• Effectiveness: match between results and Comenius objectives• Impact / Sustainability on teachers, decision-makers, education system
Inspire Summer School, Salzburg, 18-20 August 2009
Two parts of the INSPIRE External evaluation
• a) Quality of the project, consortium and activities / outcomes, its organisation
• b) Quality of the educational activities / outcomes and their (possible) impact on science education
Inspire Summer School, Salzburg, 18-20 August 2009
External evaluation tools of b) pedagogical activities
• Protocol of experimentation: data collection– Questionnaires for:
– Teachers– Pupils– National coordinators– School coordinators
– Informal interviews with stakeholders
• Analysis of the data• Participation in meetings• Draft report plus feedback from stakeholders
Inspire Summer School, Salzburg, 18-20 August 2009
Results of the external evaluation• An external evaluation report• Parts of the report
– Description of aims, methodology– Evaluation of functioning, organisation of
partnership– Evaluation of the pedagogical activities
• European Report• 5 Country reports
Inspire Summer School, Salzburg, 18-20 August 2009
Learning objects in MST teaching and learning
Presentation of the European evaluation report
of the pedagogical activities
Inspire Summer School, Salzburg, 18-20 August 2009
Part 1:Participation in the project
Inspire Summer School, Salzburg, 18-20 August 2009
Types of schools involved
Inspire Summer School, Salzburg, 18-20 August 2009
Characteristics of the schools
Inspire Summer School, Salzburg, 18-20 August 2009
Characteristics of the schools
Inspire Summer School, Salzburg, 18-20 August 2009
Teachers participating in the survey
Inspire Summer School, Salzburg, 18-20 August 2009
Experience of teachers in experiments
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils participating in the survey
Inspire Summer School, Salzburg, 18-20 August 2009
Age of respondent pupils
Inspire Summer School, Salzburg, 18-20 August 2009
Questions
• Still confusion about number of Austrian schools. How many exactly?
• Nature of VET-schools: secondary or adult education?
• Reasons for drop-out of German and Austrian schools?– Please contact Magda or Yves during summer
academy!
Inspire Summer School, Salzburg, 18-20 August 2009
Part 2: Organisation of project activities
Inspire Summer School, Salzburg, 18-20 August 2009
Part 2: Organisation of project activities
• Overall thorough preparation /information• Coordinating school teachers informed about
the project most refer to one (training) meeting or several meetings.
• The national coordinators advised the teachers as to the LO to be used through e-mail, meetings, oral and written guidelines
• Some NC indicated which LO could be used at the different levels of the schools.
Inspire Summer School, Salzburg, 18-20 August 2009
Training of teachers and pupils
Inspire Summer School, Salzburg, 18-20 August 2009
Training of teachers and pupils
• trainings of pupils lasted between 15 minutes and 10 hours
• trainings for pupils given by a number of people involved such as ICT teachers, the school coordinator, the physics teacher etc.
• training lasted between two hours and twenty hours.
• trainings were given by national coordinator or the school coordinator.
Inspire Summer School, Salzburg, 18-20 August 2009
Organisational facilitators
Inspire Summer School, Salzburg, 18-20 August 2009
Evaluation tools teachers
Inspire Summer School, Salzburg, 18-20 August 2009
Evaluation tools pupils
Inspire Summer School, Salzburg, 18-20 August 2009
Coordination of evaluation activities: key =coordinator
Inspire Summer School, Salzburg, 18-20 August 2009
Strong appreciation of School Coordinator
Inspire Summer School, Salzburg, 18-20 August 2009
Strong appreciation of National Coordinator
Inspire Summer School, Salzburg, 18-20 August 2009
Evaluation of monitoring
Inspire Summer School, Salzburg, 18-20 August 2009
Any remarks ?
• Concerning preparation or information?• Concerning training?• Concerning monitoring and support?
– KEY role of national coordinators– KEY role of school coordinators
Inspire Summer School, Salzburg, 18-20 August 2009
Part III: Implementation
• Selection criteria of LO• Use of LO per category and per subject• Use of LO for extracurricular activities
Inspire Summer School, Salzburg, 18-20 August 2009
Selection criteria of LO
Inspire Summer School, Salzburg, 18-20 August 2009
Use of LO per subcategory
Inspire Summer School, Salzburg, 18-20 August 2009
Most popular LO per subcategory
• Biology: – B2-5-26 , “Habitat of animals” used by 26
teachers, B3-5-27, the “Virtual frog builder”, 23 teachers
• Chemistry– C3-5-15, “Periodic table tetris” used 23 times and
LO C4-5-16, “Characteristics of materials, used 17 times
Inspire Summer School, Salzburg, 18-20 August 2009
Most popular LO per subcategory
• Mathematics– LO M2-5-38, “Maya numbers” used 23 times and LO
M2-13-42 “Trigonometry” used 21 times.
• Physics– LO P4-5-4, “Conservation of energy” used 22 times LO
P2-17-10, “Forces of the pendulum” 21 times.
• IT– T3-13-55, “Computer parts hangman” 33 times and LO
T4-13-56 “Click and learn computer parts” selected by 30 teachers
Inspire Summer School, Salzburg, 18-20 August 2009
Disciplines where LO were used
Inspire Summer School, Salzburg, 18-20 August 2009
Extracurricular activities where LO were used
Inspire Summer School, Salzburg, 18-20 August 2009
How did pupils work?
Inspire Summer School, Salzburg, 18-20 August 2009
Average number of LO used per pupil
Inspire Summer School, Salzburg, 18-20 August 2009
How were LO used?
• Average groups of around 20 pupils• Average time spent on LO : 1.5 – 2 hours
– Ranging from 15 minutes (languages) to 14 hours
• Mostly Inquiry-Based Science Education (IBSE) approach
• 88% of the teachers worked with special equipment– Usually computer room (sometimes with Internet)– Sometimes beamer
Inspire Summer School, Salzburg, 18-20 August 2009
Questions
• How is it possible to use LO without special IT equipment?
• What is meant by working for the LO 2 weeks, 3 months? Repeated use of same LO?
Inspire Summer School, Salzburg, 18-20 August 2009
Part IV: Impact• On teachers
– Expected impact on teaching before use of LO – Impact on teaching after use of LO– Impact on pupils’ competences– Impact on pupils’ attitudes
• On pupils– Impact of MST before use of LO– Impact of MST after use of LO
• On schools• On education system
Inspire Summer School, Salzburg, 18-20 August 2009
Teachers: Expected impact of LO
Inspire Summer School, Salzburg, 18-20 August 2009
Teachers: Impact after use of LO
Inspire Summer School, Salzburg, 18-20 August 2009
Teachers: Impact on teaching before and after
Inspire Summer School, Salzburg, 18-20 August 2009
Teachers: Impact of LO on pupils’ appreciation of MST
Inspire Summer School, Salzburg, 18-20 August 2009
Teachers: Impact on pupils’ attitudes
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils: Impact of MST before use of LO
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils’ appreciation of MST - gender
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils: Impact of MST after use of LO
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils: Impact of MST after use of LO – gender
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils: Impact of MST after use of LO – gender and age
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils: Impact of MST after use of LO – gender and age
Inspire Summer School, Salzburg, 18-20 August 2009
Pupils: Impact of MST after use of LO youngest pupils– gender
Inspire Summer School, Salzburg, 18-20 August 2009
Impact of LO: teachers vs. pupils
Inspire Summer School, Salzburg, 18-20 August 2009
Impact of Inspire project on schools
Inspire Summer School, Salzburg, 18-20 August 2009
Impact of Inspire project on education system
Inspire Summer School, Salzburg, 18-20 August 2009
Impact of project: dissemination
Inspire Summer School, Salzburg, 18-20 August 2009
Impact of project: dissemination
Inspire Summer School, Salzburg, 18-20 August 2009
Impact of project: dissemination
Inspire Summer School, Salzburg, 18-20 August 2009
Questions?
• What affected the impact most?• How could the impact be increased?
Inspire Summer School, Salzburg, 18-20 August 2009
Part V: Success factors and facilitators
Inspire Summer School, Salzburg, 18-20 August 2009
Differences
Some results from the country chapters
Inspire Summer School, Salzburg, 18-20 August 2009
Expectations
Inspire Summer School, Salzburg, 18-20 August 2009
Inspire Summer School, Salzburg, 18-20 August 2009
Inspire Summer School, Salzburg, 18-20 August 2009
Conclusions
• The Inspire project has been a success thanks to:– Motivation and hard work of all the teachers involved– Motivation and hard work of NC– Motivation and hard work of SC– Motivation of the pupils involved– The organisational structure (NC, SC etc.) set up by
European Schoolnet– The involvement of several ministries (official status
of project)
Inspire Summer School, Salzburg, 18-20 August 2009
Conclusions• The use of LO definitely enhances the quality of teaching
and learning of MST • The use of LO enhances the acquisition of LLL skills• The impact of the LO is overall higher on boys than on
girls (exception LT)• The younger the pupils are the higher the impact• The number of LO used does not affect the impact• The impact of the project is not only experienced on the
schools involved but also on the education systems of the countries involved.
Inspire Summer School, Salzburg, 18-20 August 2009
Recommendations
• Use LO much more widely across the curriculum– Selection of LO should not only be per
subcategory but also per age group– Use LO as much as possible with younger age
groups as they seem to have most impact with them
• Disseminate outcomes of Inspire project to the members of the MST Cluster of DG EAC
Inspire Summer School, Salzburg, 18-20 August 2009
Thank you for your attention!
Any more questions or remarks?Any supplementary information is welcome!