Positioning evaluation as responsive to influences and as being influential

POSITIONING EVALUATION AS RESPONSIVE TO INFLUENCES AND AS BEING INFLUENTIAL

Cheryl Ballantyne

NSW Department of Education and Communities

Western Sydney Region

[email protected]

Australasian Evaluation Society International Conference

29 August – 2 September 2011

Hilton, Sydney, Australia

AIMS OF PRESENTATION

To explore the notion of evaluation as an intervention by:

Describing a current evaluation in progress

Identifying influences to date

Outlining future directions for the evaluation

THEORETICAL CONTEXT Evaluation as an intervention has an end

goal of social betterment (Henry, 2005; Henry and Mark, 2003)

Evaluation as an intervention has potential to influence at individual, interpersonal and collective levels (Henry and Mark, 2003)

Integrated theory of evaluation influence incorporates three dimensions, source of influence, intention and time frame (Kirkhart, 2000)

KEY CLAIM

Evaluation, when incorporated as a regular intervention within program implementation can increase the program’s effectiveness and its responsiveness to contextual influences.

STRUCTURE

1. Description of program

2. What have been the contextual features that have influenced the program development and implementation?

3. What influences has the evaluation had to date?

STRUCTURE

4. Why has this evaluation had these influences?

5. How was the evaluation data generated?

6. What are the implications for developing future evaluation instruments?

1. WESTERN SYDNEY NINGBO CHINESE VOLUNTEER TEACHER-RESEARCHER PROGRAM

Partnership involving DEC Western Sydney Region, Ningbo Municipal Education Bureau, PRC, and University of Western Sydney

PROGRAM AIM

To promote the teaching and learning of Chinese language and culture in WSR schools by generating interest in schools and embedding that interest through a sustained program of support

Long term – development of students’ intercultural competence in Chinese

PROGRAM FEATURES2008-2012 18 participating schools, 10 primary and 8 high

schools selected by expression of interest, majority of students are non-background speakers of Chinese

Up to 10 graduates from Ningbo, China arrive at the end of June each year

Volunteer in schools two days per week to support the teaching and learning of Chinese language, culture and society

Additional human resource to participating schools equal to approximately $25,0000 per year

PROGRAM FEATURES2008-2012

Volunteers complete Master of Education (Honours) at University of Western Sydney

Research informs and is informed by school experience

Return to China in December the following year

2. CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION

Globalisation

Expansion of China in the worldApprox 1.2 billion English speakers in China

Approx 700 billion Chinese speakers in other countries

Implications for students’ future communities and employment

CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION

Commonwealth Government commitment to increasing Asia literacy of Australian students

2020 target

National Curriculum Languages learning area

Mandarin Chinese to be one of two target languages to be taught in NSW

Across curriculum dimension – Asia and Australia’s engagement with Asia

CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION

NSW DEC memorandum of understanding with China

NSW Confucius Institute in partnership with China

Confucius classroom in Western Sydney Region

CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION

Western Sydney Region memorandum of understanding with Ningbo Municipal Education Bureau

annual cultural exchange visits sister schools

EVALUATION RESPONSE 2010 – 2012 evaluation of Volunteer

Teacher-researcher Program to inform ongoing program development and future directions

Focus question: What can a Chinese native-speaker volunteer program contribute to the teaching and learning of Chinese language and culture in schools?

ANALYSIS OF EVALUATION INFLUENCES:

Individual

Interpersonal

Collective (Henry and Mark, 2003)

3. WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?

Individual

Increased the importance individual principals and regional staff attribute to the Program

Renewed individual principals’ commitment to the Program

Shaped Regional leaders’ attitudes towards importance of Chinese language learning and the partnership with Ningbo

WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?

Interpersonal

Principals share ideas about their Chinese programs with one another

Principals emphasise importance of Chinese programs to staff and community

WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?

Interpersonal

Additional schools have expressed interest in participating in the Program

Inquiries about/interest in the Program from interstate and national levels

Meeting of partners to be held in September 2011 to discuss extension of partnership

WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?

Collective Signing of new Memorandum of Understanding

planned for late September 2011

Increased allocation of funding to program

Regional planning to target outreach programs delivered through Confucius Classroom to high school learners

Planning of professional learning to focus on areas for further program development

EVALUATION INTERVENTION AND SOCIAL BETTERMENT In 21st century Australia, Asia literacy and

development of intercultural competence in Chinese has potential to lead to social betterment (increase students’ world view, improve employment prospects).

Evaluation has identified student, parent and teacher interest in Asia literacy through Chinese

Evaluation has revealed that students, parents and teachers regard Asia literacy through Chinese as important

Evidence has influenced further evaluation strategies, to enable a deeper analysis of influence

4. WHY HAS THIS EVALUATION HAD THESE INFLUENCES? (EVIDENCE)

0

1

2

3

4

5

6

7

Schools with Mandarin teacher

Schools with volunteer only

Schools with other support

Number of WSR primary schools providing Chinese language/culture programs by type

of support

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? (EVIDENCE)

480

500

520

540

560

580

600

620

640

Year K Year 1 Year 2 Year 3 Year 4 Year 5 Year 6

Primary students in WSR learning Chinese language/culture

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRIMARY SCHOOL SURVEYS

702 student responses from eight schools

67% - learning Chinese language important 80% - learning Chinese language interesting 63% - learning about China important 78% - learning about China interesting 65% would like to continue learning Chinese

in 2011

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRIMARY SCHOOL PARENT SURVEYS

117 parent responses from seven schools

More than 80% - learning Chinese language and culture important and valuable for students

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? HIGH SCHOOL SURVEYS

245 student responses from five schools

178 student responses from one school only

10 parent responses all from one school

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? (EVIDENCE)

0

100

200

300

400

500

600

700

800

Year 7 Year 8 Year 9 Year 10 Year 11 Year 12

Secondary students in WSR studying Chinese language courses

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? STAFF SURVEYS

50 staff responses representing 12 schools38 from 6 primary schools12 from 6 high schools

95% - learning Chinese language important

100% - learning Chinese culture important

93% - Chinese Program has a positive influence on schools

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? STAFF SURVEYS

0

5

10

15

20

25

30

30 minutes to one hour one to two hours more than two hours

Time spent on Chinese language/culture per week

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRINCIPAL FOCUS GROUPS

Volunteers’ rapport with students has stimulated interest in Chinese language

Chinese language is accepted as part of the curriculum (no longer exotic – High School Principal)

Planning for Chinese language and culture programs occurring in primary schools and high schools

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRINCIPAL FOCUS GROUPS

Chinese language has replaced other language programs - two high schools

Links between Chinese language and culture and other learning areas - three high schools

Chinese classroom allocated - one high school

WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRINCIPAL FOCUS GROUPS

Collaboration across learning communities to support transition Year 6 to 7.

System support important to the long term effectiveness of school programs

5. HOW WAS THE EVALUATION DATA GENERATED

DEC annual census information on languages programs

Focus group of Principals term 1 2010

Surveys of students, parents and staff distributed to 18 participating schools term 3 2010

EVALUATION INFLUENCE: SOURCE, INTENTION, TIME FRAME

Source – process (focus groups) and reporting of evidence (results)

Intention – intended (focus groups, reporting of evidence)

Time frame – immediate, evaluation embedded in program implementation

(Kirkhart, 2000)

6. WHAT ARE THE IMPLICATIONS FOR FUTURE EVALUATION INSTRUMENTS? FUTURE DIRECTIONS

Student self-assessment of intercultural competence to enable deeper analysis of Program’s impact on student learning

Parent and teacher assessment of student intercultural competence to enable deeper analysis of impact on student learning

6. WHAT ARE THE IMPLICATIONS FOR FUTURE EVALUATION INSTRUMENTS? FUTURE DIRECTIONS

School self-evaluation (based on program standards) to enable deeper analysis of school programs and influence school planning

Document analysis of school plans, policies and annual school reports to analyse extent to which Chinese programs are valued, planned for and embedded in whole school operation

CONCLUSION This evaluation (in early stages) has had

influence at school, regional, state, interstate and national levels

Further planning of evaluation instruments is intended to deepen influence

This evaluation, as an intervention embedded in program planning and implementation, has potential to make a difference for Western Sydney students

THANK YOU FROM CHERYL BALLANTYNE

FOR CONSIDERING - POSITIONING EVALUATION AS RESPONSIVE TO INFLUENCES AND AS BEING INFLUENTIAL

Australasian Evaluation Society International Conference

29 August – 2 September 2011

Hilton, Sydney, Australia