A Realist Evaluation of Performance Management for Social Services Lessons from the implementation of “Outcome Management” in the voluntary sector in Singapore.

A Realist Evaluation of Performance Management for

Social Services

Lessons from the implementation of “Outcome Management” in the

voluntary sector in Singapore

Robyn Tan, PhD candidateInstitute of Development Policy and Management

University of ManchesterEmail: [email protected]

“Public sector performance measurement is, in effect, like putting a meter on a black box: we have little knowledge of the mechanism

inside and no theory linking inputs, processes, outputs and outcomes to explain

why a particular result occurred or to prescribe what management or

organizational adjustments are needed to improve performance”

(Grenier 1997)

The core of performance management is for achieving accountability and

improvement

- Yes? No? Not sure?

Research Aim

To examine how and why performance management works (or not) to achieve accountability and improvement among voluntary organisations, through the use

of performance information

Case Example: Outcome Management for “Job Placement and Job Support”

Outcome StatementOut of 55 clients who were engaged in job placement interview, 40 stay in the job for at least 6 months.

Milestones Targets Measurement Tools

1.Clients are engaged in a job placement interview

55 Registration Client Progress Report

2. Clients are engaged in the development of employment plans

55 Programme recordEvaluation Report

3. Clients are matched to suitable jobs and attend job interviews

50 Placement Record

4. Clients accept job offer 50 Placement Record

5. Clients stay in the job for at least 2 weeks

50 Placement Record

6. Clients stay in the job for at least 3 months

42 Feedback form Site visitCase notes

7. Clients stay in the job for at least 6 months

40 Feedback formSite visitCase notes

For practitioners to meet specified targets, in order to maintain accountability

For practitioners,

(1) To learn about client’s

progress in relation

to the milestones;

(2) To improve

the programme

Applying realist evaluation to the study of performance

managementRealist evaluation Primary data sources Secondary data sources

Develop the initial programme theories of how and why Outcome Management is expected to work

NIL Policy papersTraining manualOutcome Management related literaturePerformance management literature

Test the actual theories of how and why Outcome Management works against the initial theories, using 7 identified case studies

Individual & group interviews with Council programme administrators and voluntary organisation practitioners

Social service contractPerformance reports

Develop refined programme theories of how and why Outcome Management would work

Data analysis from primary and secondary data sources

How to unpack a complex intervention such as

performance management?

Implementation chain of Outcome Management

Step 1: Identify initial, intermediate & long term outcomes

Step 2: Specify

programme in relation

to outcomes

Step 3: Select performance indicators &

measurement tools

Step 4: Set & meet targets

Step 5: Use performance information

for learning & improvement

How to identify the outcomes for the study?

“Outcome Management is intended to serve as a robust and multifaceted

oversight system for social services and to improve the overall service standards

of service delivery”

– National Council of Social Service

Developing initial progamme theories:

Identifying sub-outcomes & higher-

order outcomes

Step 1: Identify initial, intermediate & long term outcomes

Step 2: Specify

programme in relation

to outcomes

Step 3: Select performance indicators &

measurement tools

Step 4: Set & meet targets

Step 5: Use performance information

for learning & improvement

Oi: Voluntary organisations

formulate initial, intermediate

& desired outcomes

Oii: Voluntary organisations specify

prog activities or intervention strategies in relation to the outcomes

Oiii: Performance information generates

feedback on client progress in relation to

the outcomes

O1: Voluntary organisations meet

targets, in turn achieve accountability

to the Council

O2: Voluntary organisations learn

about clients’ progress

O3: Voluntary organisations make actual changes to

improve the programme

Initial programme theories & actual programme theoriesOutcome

ManagementInitial programme theories Actual programme theories

Step 1: Developing outcomes

C: Institutional requirementC: Equipped with necessary knowledge and skillsM: “Acquiescence”Oi: To formulate well-defined outcomes

C: Limited knowledge and skillsC: Intervention-specific contextM: “Mimicking” tactics M: “Compromise” tacticsO-: Fail to formulate well-defined outcomes

Step 2: Specifying

programme

C: Lack of institutional requirementC: Equipped with necessary knowledge and skillsM: “Tight coupling”Oii: To specify programme in relation to the outcomes

C: Lack of institutional requirement; depends on organisational discretionC: Intervention-specific contextC: Limited knowledge and skills M: “Decoupling” O-: Fail to specify programme in relation to the outcomes

Step 3: Adopting

measurement tools

C: Institutional requirement to measure outcomes quantitatively; choice of measurement tools based on organisational discretionC: Equipped with necessary knowledge and skills M: “Tight coupling”Oiii: To generate feedback on client progress in relation to the outcomes

C: Choice of measurement tools based on organisational discretionC: Limited knowledge and skillsM: “Decoupling”M: “Pragmatism”O: Lack of standardisation in the use of measurement toolsO: Lack of routine monitoringO-: Fail to generate feedback updates on client progress

Step 4: Setting and

meeting targets

C: Institutional requirements to meet targetsM: “Target system” M: “Ranking system”M: “Signal function”O1: To meet targets in order to achieve accountability

M: “Target system”O-: Targets were met BUT they were measured against poorly-defined outcomes

Step 5: Using

performance information

Ci: Well-defined outcomesCii: Specification of programme in relation to outcomes Cii: Generated feedback on client progress in relation to outcomesM: “Feedback” mechanismO2: To learn about clients’ progress in relation to outcomesO3: To make actual changes for improvement

Ci-: Poorly-defined outcomesCii-: Fail to specify programme in relation to outcomesCiii-: Fail to generate feedback on clients’ progressO2-: Fail to learn about clients’ progressO3-: Fail to make actual changes for improvement

Key findings

Of the 7 case study programmes, only 1 achieved accountability and improvement

Lack of institutional requirement for voluntary organisations to articulate and test the underlying theory of change (institutional context)

Outcome Management is applied to the measurement of outcomes of a wide array of social services without considering the programme-specific characteristics (intervention-specific context)

Key findings (II)Programmatic intervention Single practitioner-driven

interventione.g. Job placement and job support; sheltered workshop

e.g. Counselling; case management

Reasonably standardised programme across practitioners

Individualised interventions based on practitioner’s choice of theoretical approach and needs of clients

One overarching theory of change There are as many theories of change as there are practitioners and clients

Practitioners deliver programme according to the articulated theory of change

Practitioner delivers intervention based on his or her own theory of change (whether articulated or not).

Facilitates learning and improvement across practitioners

Learning and improvement (if any) is based on practitioner discretion

Developing refined programme theories

Step 2: Identify initial,

intermediate & long term outcomes

Step 3: Specify programme in

relation to outcomes

Step 4: Select performance indicators &

measurement tools

Step 5: Set & meet targets

Step 6: Use performance

information for learning &

improvement

Step 1: Articulate theory of change

underlying programmatic

intervention

Reflections

Identifying the level of abstraction for data collection and analysis – higher or lower or at what level?

Identify the level of abstraction that can provide relevant theoretical and practical insights required for the study

Reflections (II)

Identifying mechanisms from contexts – which is which? And does it matter?

e.g.

“The Council gives voluntary organisations discretion over choice of measurement tools” (context)

“Voluntary organisations exercising their discretion for their choice of measurement tools” (mechanism)

Identify the context-mechanism dyad responsible for generating the outcome

Reflections (III)

Making sense of the contexts for the development of refined programme theories

Categorising contexts according to their layers of social reality – individual capabilities, organisational context, institutional context, intervention-specific context

Identifying the “order” in which they are likely to work to give the intervention a reasonable chance of success.

Thank you