Developing a Monitoring and Evaluation Framework: Insight into internal stakeholder learnings Beth Ferguson AES Conference Sydney 2 September 2011.

Developing a Monitoring and Evaluation Framework: Insight into internal stakeholder learningsBeth Ferguson

AES Conference Sydney 2 September 2011

Outline

• Background and context• Monitoring and evaluation framework and

tools • Implementation of the framework• Stakeholder experiences and learnings

Background

• Immigration New Zealand funds a variety of services to help new migrants and refugees to New Zealand to settle in to their new home

• $5m of funding towards 30 contracts• 18 services are Settlement Support New Zealand

(SSNZ) initiatives• 3 skills matching programmes

Background

• In 2008, a monitoring and evaluation of settlement services was initiated.

• Part of a government commitment to: ‘strengthen settlement services by establishing a robust

evaluation process to ensure effectiveness’

Purpose

• Purpose was to develop a strategy for a monitoring and evaluation framework to:• determine the overall quality or value of services the

Department funds• contribute to the process of continuous improvement

for existing services• Ongoing monitoring and evaluation rather than a

one-off evaluation

Rationale

• Developed to monitor immediate outcomes of settlement services funded by the Department of Labour to:• assist with self-learning and self-assessment • make criteria explicit and information needs transparent• be integral to the business and contract management• enable immediate outcomes data to be collected, and• ensure sustainability over time and minimal

requirements for service providers.

Stakeholders

• Implementation & Development: • Migration Research

• For use by:• Settlement Purchasing Team at the Department of

Labour• Providers of settlement services funded by the

Department

The Framework Tools

• Good Practice Matrix

• Service logics• Provider feedback • Surveys

Mid-year & Full year reports

Good Practice Matrix

• Based on an international literature review of good practice

• Will be modified/updated as new evidence becomes available

• Helps services to identify areas of strength and areas to potentially focus on

Assesses alignment with Good Practice Principles:• Infrastructure• Targeting • Migrant Involvement• Integration of services• Service Delivery

Across 3 levels:• Preferred practice• Good practice • Acceptable practice

Good Practice Matrix cont…

Service logics

• Used as a way to visually and logically clarify the links between inputs, activities, outputs and outcomes of the contracted settlement services.

• Logics will be used as a tool when contracting new settlement services

• Tool for services to map progress towards Key Performance Measures

Provider Feedback

• A tool for contracted settlement service providers to reflect on the work of their service and the delivery of the service against the contract.

• Service providers report on:• what is working well for their service • What is not working well• Improvements that need to be made

Surveys

Client Feedback Survey• Designed to monitor trends and client outcomes• Based on the Common Measurement Tool (CMT)• Translated into 8 languages

(Chinese, Korean, Spanish, Russian, Hindi, Tagalog, Japanese, Arabic)

Surveys

Local Settlement Network (LSN) Survey• For use by Settlement Support New Zealand

(SSNZ) initiatives throughout the country

• Designed to assess how well SSNZ initiatives are working with local agencies to increase their responsiveness to the needs of newcomers

Reports

• Reports have been developed to summarise information from the monitoring and evaluation tools and to provide overall benchmarks

Mid-year reports:• Produced in February to update on progress

Full-year reports:• Comprehensive report produced in August• Include all monitoring and evaluation tools

Implementation

• Trialled with 9 settlement service providers in May 2010

• Some additions and refinements were made as a result of the trial

• Implemented November 2010 with 27 settlement services

Stakeholder experiences: Researchers/Evaluators

• Challenging process • Valuable to up-skill/train others to use the tools in a

consistent way• Utilising surveys as an educative tool has been

useful – has provided an opportunity to inform clients about the purpose of the service.

Stakeholder experiences: Settlement Purchasing team

• Practicalities translating a framework developed using the literature into an operational tool

• Need for ongoing support from the developers of the Framework

• need for ongoing funding to support the framework.

Stakeholder experiences: Settlement service providers

• Feedback predominantly positive• Appreciative of transparency • Keen to know how they can improve• Some questions about the implementation• Some concerns about use of results• Elements of the surveys in particular can be labour

intensive

Joint Learnings

• Need a common understanding from the beginning about:• The problem and assumptions • What is to be achieved from the framework.

• Need consultation and involvement in the development of the Framework by all stakeholders

• Need to discuss and agree on the viability of different approaches

Joint learnings

• Engagement with service providers• Trial and pilot tools• Training and support to those using the tools• Use a tried and tested survey (CMT)• Translations – complex and time consuming

Conclusions

• The Department has successfully developed and implemented an ongoing monitoring and evaluation framework

However …..• Developing and implementing a monitoring and

evaluation framework is complex:• Variety of stakeholders • Differing backgrounds • Differing understandings of evaluation and purpose

For more information….

• Email:Beth.Ferguson@dol.govt.nz

• Migration research website:http://dol.govt.nz/research/migration/index.asp