STORIES FOR CHANGE: A NEW MODEL OF EVALUATION FOR ... · STORIES FOR CHANGE: A NEW MODEL OF EVALUATION FOR AGRICULTURAL EXTENSION PROJECTS IN AUSTRALIA Jessica Jane Dart Submitted

STORIES FOR CHANGE:

A NEW MODEL OF EVALUATION FOR AGRICULTURAL

EXTENSION PROJECTS IN AUSTRALIA

Jessica Jane Dart

Submitted in total fulfilment of the requirements of the degree of Doctor of

Philosophy

July 2000

Institute of Land and Food Resources, the University of Melbourne

i

ABSTRACT

A model of evaluation was investigated and developed to help meet the evaluation needs of

agricultural extension agencies in Australia. The model was based on Davies’ (1996) Most

Significant Change (MSC) model; a participatory evaluation approach that involves the generation

of stories and their interpretation by project stakeholders. This model had not previously been

implemented in Australia.

Over the last decade agricultural extension institutions have been undergoing structural and

philosophical changes. These changes have brought about an increased demand for evaluation that

can deal with participatory, group-based extension, and can identify and value diverse impacts.

There is also an increased demand for internal evaluation that can deal with multiple audiences,

promote organisational learning and communication, as well as meet needs for accountability. To

develop an evaluation approach capable of addressing these needs, a novel evaluation model was

selected, modified, implemented and empirically tested.

A case evaluation of the MSC approach was conducted across a large dairy extension project

operating in four regions of Victoria. After implementing the evaluation, a meta-evaluation was

conducted to determine the extent to which the MSC process contributed to project improvements,

met needs and represented ‘good’ evaluation. It was found that it met the articulated needs of the

stakeholders of the case project and combined well with their existing evaluation practices. This

was ultimately verified by their commitment to continue with the MSC process after the 12 month

trial period. Project staff perceived that the MSC process helped stakeholders to make sense of

project impact and to understand each other’s values, increased staff morale and helped draw staff,

farmers and other collaborators more centrally into the evaluation process. However, there were

regional differences in the perception of benefits achieved. An examination of these differences

provided information about the sort of project contexts in which benefits are likely to be gained

from the MSC model.

In considering the evaluation needs of extension more broadly, the MSC model was found to offer

processes capable of accommodating the new genre of participatory extension projects. Where

projects are likely to have diverse outcomes that are not all pre-determined, the MSC model could

ii

play an important role in searching for impact. The MSC model can also accommodate a wide

range of participants in the evaluation process. However, the model does not provide indication of

the spread of adoption of technologies across the farming population, so does not fully meet the

demand by purchasers for greater accountability. It was designed to capture ‘remarkable events’

rather than the average experience of farmers. While the MSC model satisfied some of the premises

for good evaluation, it was found to have inherent biases and weaknesses, including an inability to

capture unintended outcomes unless these are deemed ‘significant’ by those involved in the process.

It was suggested that the MSC model should be strategically combined with other evaluation

approaches selected to offset its inherent bias, and to meet the full range of project needs for

evaluation.

The MSC model was found to offer an important contribution to the ‘basket of choices’ for

extension evaluation, but like other evaluation models, it does not provide an overarching solution

to the myriad of evaluation demands being felt by extension agencies in Australia. It is suggested

that those evaluating extension projects should first come to understand the evaluation needs of

their project stakeholders, and then develop a carefully crafted collection of evaluation approaches

to meet these requirements. For some projects, the MSC model may provide an important new

component of their evaluation strategy. In particular, large, process-orientated projects, with high

levels of farmer involvement, could gain much from this model.

iii

This is to certify that

(i) the thesis comprises only my original work except where indicated in the preface,

(ii) due acknowledgment has been made in the text to all other material used,

(iii) the thesis is less than 100,00 words in length, exclusive of tables, maps, bibliographies and

appendices.

Jessica Jane Dart

iv

PREVIOUS PUBLICATION OF MATERIAL FROM THESIS

Earlier versions of the review of evaluation in agricultural extension presented in Chapter 4 were

published in the following documents and conference papers:

Dart, J.J., Petheram R.J., & Straw, W. (1998), Review of Evaluation in Agricultural

Extension, Rural Industries and Research Development Corporation, publication number

98/136, Canberra.

Dart, J.J., Bardsley B., & Petheram R.J., (1997), ‘Forms and approaches of evaluation in

agricultural extension’, Proceedings of the Australasian Evaluation Society (AES)

International Conference, Adelaide, AES, 210-225.

Dart, J.J., Petheram R.J., & Straw, W. (1997), ‘Forms of evaluation – for agricultural

extension’, Proceedings of the 2nd Australasia Pacific Extension Conference (APEN),

Albury, NSW, APEN, 2, 408-471.

Earlier versions of the description concerning the implementation of the MSC model, presented in

Chapter 7, were published in the following papers:

Dart, J.J. with Dysdale G., Cole, D., Saddington, M. (in press), ‘The most significant

change approach for monitoring an Australian extension project’, PLA Notes: Participatory

Learning and Action, International Institute for Environment and Development, London.

Dart, J.J. (1999), ‘A Story Approach for monitoring change in an agricultural extension

project’, proceedings of the Association for Qualitative Research (AQR), International

Conference, Melbourne, AQR, world wide web:

http://www.latrobe.edu.au/www/aqr/offer/papers/JDart.htm

Dart, J.J. (1999), ‘The tale behind the performance story approach’, Evaluation News and

Comment, 8, no.1, 12-13.

v

ACKNOWLEDGEMENTS

Firstly, I would like to acknowledge all the effort and help that has been provided by my

supervisory panel. A big ‘thank you’ to Dr Patricia Rogers for stepping in during the final year and

providing excellent guidance and expertise on program evaluation and also for encouragement and

support. Many thanks to Dr John Petheram, who has been involved with this research from the very

beginning, and who has been a great help. I would also like to acknowledge important contributions

from Dr Ruth Beilin and Dr Barrie Bardsely.

I received valuable and important help from many staff of the Victorian Department of Natural

Resources and Environment. Thanks to Bron McDonald for being my mentor throughout the three

years, and for sharing her expansive networks with me. I am also very grateful to all the staff and

committees members of the Target 10 Dairy Extension Project, who took the risk of participating in

this somewhat unconventional research. I would especially like to thank John Boomsma, Mark

Walton, David Cole and Geoff Drysdale for their help and support in implementing the Most

Significant Change evaluation process. I am also grateful to all the storytellers for contributing their

stories to the evaluation process.

I would like to acknowledge several people for sharing with me their wisdom and advice

concerning program evaluation. Thanks to Rick Davies for his pioneering work with the MSC

model, and for insightful conversations. The staff from the Centre for Program Evaluation at the

University of Melbourne have also been a great help, especially Dr Gerald Elsworth. Thanks also

to Jerome Winston from RMIT University for his inspirational ideas concerning program

evaluation.

Finally, I am thankful for the help of my friends and family for being patient and accepting. Thanks

to Scott for help with the technical aspects of the Internet questionnaire, and thanks to Carolyn for

putting up with me and sharing an office space through these three years. Thanks to Mick Maguire

for helping me with the production of the booklet Target 10 Evaluation Stories.

vi

ACRONYMS

CBA Cost benefit analysis

CCBD Christian Commission for Development in Bangladesh

CEC Central Executive Committee

CSIRO Commonwealth Scientific and Industrial Research Organisation

DPI Department of Primary Industries (Queensland)

DRDC Dairy Research and Development Corporation

FGE Fourth Generation Evaluation (Guba and Lincoln, 1989)

IIED International Institute for Environment and Development

KASA Knowledge, attitudes, skills and aspirations

LWRRDC Land and Water Resources Research and Development Corporation

M&E Monitoring and evaluation

MIA Murrumbidgee Irrigation Area

MSC Most Significant Change model (Davies, 1996)

NPM New public management

NRE Natural Resources and Environment

NSCP National Soil Conservation Program

NUD.IST Non-numerical Unstructured Data Indexing Searching and Theorizing

OECD Organisation for Economic Development and Cooperation

ORID Objective, Reflective, Interpretive, Decisional

PAR Participatory Action Research

PEMS Pig Enterprise Management Services

PM&E Participatory Monitoring and Evaluation

PRA Participatory Rural Appraisal

PPRDP People’s Participatory Rural Development Program

RIRC Rural Industry Research Corporations

RIRDC Rural Industry Research and Development Corporation

RRA Rapid Rural Appraisal

UFE Utilization Focused Evaluation (Patton, 1997)

VFSG Variable Farming Systems Group

vii

CONTENTS

CHAPTER 1: INTRODUCTION

1.1 Problem Statement 1

1.2 Aim 2

1.3 Research Objectives 3

1.4 Research Approach 3

1.5 Road Map for Thesis 5

1.6 Overview of Chapters 7

CHAPTER 2: A CONCEPTUAL FRAMEWORK FOR GOOD PROGRAM EVALUATION

2.1 Introduction 10

2.2 What is Program Evaluation? 11

2.3 What is ‘Good’ Program Evaluation? 18

2.4 Six Normative Approaches to Evaluation 19

2.5 Bricolage and Mixed-Approach Designs 31

2.6 A Normative Framework for Evaluation 34

2.7 Conclusions 37

CHAPTER 3: EVALUATION NEEDS IN AGRICULTURAL EXTENSION

3.1 Introduction 38

3.2 A Changing World 39

3.3 Changing Philosophy of Agricultural Extension 40

3.4 Organisational Change and Extension Evaluation 52

3.5 A Case Study of the Agriculture Division in the Victorian Department of Natural Resources

and Environment 57

3.6 Conclusion 65

CHAPTER 4: REVIEW AND DESCRIPTION OF CURRENT PRACTICE OF EVALUATION IN

AGRICULTURAL EXTENSION

4.1 Introduction 67

4.2 Background to the Review 68

viii

4.3 Approach Taken 70

4.4 Description of the Evaluation Studies 73

4.5 Case Studies of Different Forms of Evaluation in Agricultural Extension 75

4.6 Summary of the Evaluation Samples 97

4.7 Examining the Evaluation Studies Against the Meta-evaluation Questions 99

4.8 Conclusions 105

CHAPTER 5: CONTENDING MODELS OF PROGRAM EVALUATION FOR AGRICULTURAL

EXTENSION

5.1 Introduction 107

5.2 Criteria for Choosing a Model to Pilot 108

5.3 Contending Models of Program Evaluation to Test 110

5.4 Utilization Focused Evaluation (UFE) 111

5.5 Realistic Evaluation 115

5.6 Fourth Generation Evaluation (FGE) 122

5.7 Most Significant Change (MSC) Model 127

5.8 Selection of Evaluation Model to be Tested 133

5.9 Considering all Four Models 138

5.10 Conclusions 139

CHAPTER 6: METHODOLOGY FOR THE CASE EVALUATION


6.2 The Overarching Research Design 141

6.3 Rationale for Choice of Project 143

6.4 Participatory Action Research Component 144

6.5 Meta-evaluation 148

6.6 Ethical Considerations 160

CHAPTER 7: CASE EVALUATION OF THE MSC MODEL AS IMPLEMENTED ACROSS THE TARGET 10

PROJECT


7.2 Description of the Project 162

7.3 Implementation of the Approach 168

7.4 Analysis of Outputs 181

7.5 Problems During the Implementation 198

7.6 Conclusions Concerning the Implementation of the MSC Process 200

ix

CHAPTER 8: THE IMPACT OF THE MSC PROCESS ON THE TARGET 10 DAIRY EXTENSION PROJECT


8.2 Method of Testing the CMO Configurations for the MSC Process 202

8.3 Different Contexts 206

8.4 Findings 209

8.5 Summary of Testing Model 238

8.6 Impact of the MSC Process on the Project 239

8.7 Conclusions 243

CHAPTER 9: THE MOST SIGNIFICANT CHANGE MODEL FOR EVALUATION IN AUSTRALIAN



9.2 Applying the Meta-Evaluation Questions 245

9.3 Is the Evaluation Relevant to the Context and Purpose in Hand? 246

9.4 To What Extent Can the MSC Evaluation Contribute to Projects Better Able to Meet Needs? 250

9.5 Is the Evaluation Model Guided by Program Theory? 251

9.6 Does the Evaluation Model Make an Attempt to Account for Different Program Outcomes? 254

9.7 Are the Evaluation Processes Socially and Politically Just? 255

9.8 Is there an appropriate ratio of Costs and Benefits for the Project from the Process

(and Outcomes) of the Evaluation? 257

9.9 To what Extent are the Design and Data Collection/ Analysis of the Evaluation Valid? 257

9.10 Does the Evaluation Attempt to Judge the Merit and Worth of Projects? 260

9.11 Summary: Extent to which the MSC Model adds to the ‘Basket of Choices’ for Extension

in Australia 262

9.12 Improving the Contribution of the MSC Model 262

CHAPTER 10: CONCLUSIONS 265

REFERENCES 269

APPENDICES 284

x

Figures

Figure 1’Road Map’ of layout of thesis and research.....................................................................................................6

Figure 2 Outcomes and intermediate outcomes of six different normative approaches to evaluation..........................30

Figure 3 The process of modifying an evaluation bricolage for a mega-project..........................................................33

Figure 4 Example of an evaluation bricolage for an hypothetical program, showing three domains ..........................73

Figure 5 Distribution of the evaluation studies according to Owen’s five forms .........................................................74

Figure 6 Graph of net present value at different adoption rates....................................................................................79

Figure 7 Distribution of evaluation studies according to the level of investigation of Bennett’s Hierarchy ..............102

Figure 8 Level of participation in evaluation by farmer clients in the studies............................................................104

Figure 9 The realist evaluation cycle..........................................................................................................................116

Figure 10 Diagram to represent monthly flow of stories in MSC model for a hypothetical project ..........................129

Figure 11 A systems diagram to illustrate case-study design including concurrent action research and meta-

evaluation components .................................................................................................................................142

Figure 12 An evaluation theory model showing how the MSC process was thought to bring about

improvements in the project .........................................................................................................................155

Figure 13 The service delivery model of the Target 10 Dairy Extension Project.......................................................163

Figure 14 Items contained in the proforma for collection of stories...........................................................................170

Figure 15 Proposed main steps for implementation of MSC process with Target 10 Project ....................................176

xi

Tables

Table 1 Owen’s five forms of program evaluation........................................................................................... 72

Table 2 Bennett’s Hierarchy............................................................................................................................. 85

Table 3 Summary of main characteristics of the sample of evaluation studies ................................................ 97

Table 4 Results of the Delphi process used to establish domains of change .................................................. 172

Table 5 Questions asked to help describe the stories...................................................................................... 182

Table 6 Number of stories collected each round ............................................................................................ 183

Table 7 Distribution of stories per region....................................................................................................... 183

Table 8 Distribution of the story tellers.......................................................................................................... 184

Table 9 Project ‘programs’ mentioned in the story ........................................................................................ 184

Table 10 Description of themes that have been the subject of multiple stories.............................................. 185

Table 11 Level of Bennett’s Hierarchy .......................................................................................................... 186

Table 12 Grouping of stories by theme using Bennett’s Hierarchy................................................................ 187

Table 13 Characteristics of stories selected by the Central Executive Committee (n=24) ............................. 190

Table 14 Conjectured context, mechanism, outcome configurations for the MSC process ........................... 204

Table 15 Staff attending their regional committee meetings during the trial ................................................. 206

Table 16 Attendance at regional committee meetings during trial ................................................................. 206

Table 17 Number of stories contributed by farmers and other non-staff committee members per region ..... 206

Table 18 Extent to which meetings in the different regions provided a context conducive to discussion

and exchange of views.......................................................................................................................... 208

Table 19 Respondents’ descriptions of the stories.......................................................................................... 210

Table 20 Number of stories read in booklet against number of story sessions attended ................................ 211

Table 21 Extent to which respondents remembered the stories...................................................................... 212

Table 22 Staff perception of the extent to which the MSC process helped farmer committees to ‘steer’

the project against meeting context....................................................................................................... 214

Table 23 Staff perception of morale boosting effect of the MSC process against meeting context ............... 218

Table 24 Staff perceptions of the extent to which the MSC process affected staff understanding of project

impact against meeting context............................................................................................................. 222

Table 25 Extent to which the MSC process affected staff understanding of outcomes valued by the CEC

against meeting context......................................................................................................................... 224

Table 26 Extent to which staff felt that the MSC process had affected their understanding of the outcomes

valued by the purchasers meeting context ............................................................................................ 225

Table 27 Staff perception of the extent to which the MSC process affected the creation of a more shared

vision against meeting context.............................................................................................................. 228

Table 28 Extent to which staff used the stories .............................................................................................. 233

xii

Table 29 Extent to which staff used the stories against meeting context ....................................................... 233

Text boxes

Text box 1 Segment of discussion at a regional committee meeting during story selection ......................................179

Text box 2 Examples of comments from the purchasers concerning the stories ........................................................191

Text box 3 Story example 1........................................................................................................................................193




Text box 7 Extract of discussion between two extension staff from the South West Region, taken from a

focus group discussion between regional champions.........................................................................................218

Text box 8 Comments concerning the competitive aspect of the MSC process.........................................................219

Text box 9 Comments associated with the benefits of discussing the stories.............................................................227

Text box 10 Comments concerning the value of the MSC process for creating dialogue with purchasers................227

Text box 11 Comments associated with the value of stories for evaluation reporting ...............................................235

Text box 12 Comments referring to how stories have affected change at a strategic level ........................................236

Text box 13 Informant views of the extent to which the farmer committee members ‘represent’ the wider

dairy farming community .................................................................................................................................256

1

CHAPTER 1: INTRODUCTION

1.1 Problem Statement

In Australia agricultural extension is undergoing philosophical change. This change is influencing

how agricultural extension is conceptualised, how it is conducted, and consequently how it should

be evaluated. The science-push paradigm, that has conventionally underpinned extension, is being

partially replaced by new approaches influenced by the emergent farmer-first paradigm. This has

resulted in a movement towards participatory, group-based and process-oriented forms of extension.

These new models of extension require models of evaluation that can deal with diversity in

implementation, participation of stakeholders within evaluation, and the need to provide

accountability for projects that begin with loosely defined outcomes.

The 1990s have also seen considerable structural and policy reform across the organisations that

deliver extension. For example, as a result of restructuring and increased collaboration, large

projects have emerged which are bigger and more complex in structure than in the past. A second

example of change is that in most states of Australia, the purchaser-provider model has been

introduced for the provision of publicly-funded extension. In simplistic terms, the purchaser is

responsible for investing resources wisely, and demonstrating the value of the investment. The

provider’s role is to provide the service contracted and to assist with demonstrating the value of the

service. With the purchaser-provider split, accountability and communication have become crucial.

These are but two of a plethora of recent changes that pose considerable challenges for the

evaluation of extension projects.

Since the introduction of these changes in agricultural extension, the demand for evaluation appears

to have increased both in terms of the diversity of approaches sought, and the number of

stakeholders requesting evaluation information. Extension project managers are being asked to

provide evidence as to whether both tangible and intangible outcomes have been achieved, to

describe the expected and unexpected impacts caused by interventions and to determine the extent

to which the farmer-clients perceive that their needs have been addressed. These managers are

required to produce plans of how they intend to conduct evaluation, yet there are no clear guidelines

2

on how this should be done. At the same time, there is a lack of knowledge among extension agents

of evaluation approaches able to address these new demands. An increased demand for outcome-

focused evaluation is particularly problematic for projects that comprise participatory group-based

models of extension. These projects tend to produce diverse, context-specific outcomes, which are

difficult to aggregate in order to provide information to evaluate the project as a whole.

The literature of agricultural extension reveals very little information on approaches that can meet

these needs (Dart et al., 1998). There is a serious gap in the knowledge required to effectively

evaluate these large, complex projects. Yet, there is an extensive literature concerning evaluation in

the fields of education, health promotion and international aid and development. There appears to

be considerable scope for transfer of ideas and learning about evaluation between the different

disciplines. Madaus et al., (1983) point out that there is a need to educate evaluators in the

availability of new techniques; a need to try out and report the results of using these new

techniques; and to develop additional methods. New approaches need to be developed, or adapted

from other disciplines, to meet the evaluation needs of agricultural extension. The emphasis should

be on making the approaches fit the needs of extension agents and their organisations, and the

farmers and public whose needs they aim to meet.

1.2 Aim

The aim of this thesis is to develop a model of evaluation that has the potential, in combination with

other approaches, to address the current challenges associated with evaluating the new genre of

agricultural extension projects in Australia.

3

1.3 Research Objectives

The research objectives are to:

• examine the evaluation needs and current practice of program evaluation in agricultural

extension projects in Australia

• identify suitable models of evaluation for agricultural extension, that have not been

implemented in Australia previously

• select, implement, modify and evaluate one novel model of evaluation across an Australian

extension project

• develop an improved and context-specific model of evaluation that has potential to bring about

improved agricultural extension projects which better meet stakeholders needs

• provide a coherent strategy to combine this approach with others, in order to address the

challenges posed by changes occurring in extension organisations.

1.4 Research Approach

In order to develop or adapt evaluation approaches to meet the needs of agricultural extension, a

pragmatic stance was taken with regard to evaluation design. The current genre of large-scale

extension projects has various evaluation needs that will only be met through the application of a

range of complementary evaluation approaches and tools. The overall approach to evaluation design

advocated in this thesis can be described as ‘bricolage’; a collective term that has been used to

describe qualitative research (Denzin and Lincoln, 1998) and cultural studies (Nelson et al., 1992):

The bricoleur produces a bricolage, that is a pieced-together close-knit set of practices that provide

solutions to a problem in a concrete situation. The solution (bricolage) which is the results of the

bricoleur’s method is an emergent construction (Weinstein and Weinstein, 1991:161 cited by Denzin

and Lincoln, 1998: 3).

In this case the bricolage is the situation-specific solution to a whole range of evaluation needs and

requests that exist for one large project. The practices pieced together comprise different

approaches and methods of evaluation. In building this evaluation bricolage, therefore, it is

necessary to understand the evaluation needs. This understanding is best gained through an analysis

of the evaluation requirements of the specific organisation in conjunction with an analysis of their

4

existing evaluation practises.

In this thesis, evaluation needs were determined at two levels and both contributed to the selection

and modification of a novel evaluation model:

• evaluation needs and current evaluation practice were explored at a generic level for

agricultural extension projects in Australia

• the specific evaluation needs and current practice were identified for one large extension

project.

A novel model of evaluation was then selected on the grounds that, (a) it could help address the

evaluation needs and complement the existing portfolio of evaluation practice of the case project

and (b) it could address some of the challenges in evaluating large extension projects in general.

The empirical research consisted of implementing a single evaluation study that was to form one

part of the evaluation strategy of the Target 10 Dairy Extension Project. The selected model was

implemented and later was itself evaluated. It was evaluated for its ability to meet evaluation needs

at the project level, and to meet the challenges posed for extension evaluation at a national level.

Implementing, modifying and evaluating the model

In conducting the research for the case study implementation of a novel evaluation model, two main

research approaches were employed; Participatory Action Research (PAR) and a theory-guided

approach to meta-evaluation. Key differences between the two research components were the

degree of participation of project stakeholders and the purpose for which the information was

collected. The Target 10 Project stakeholders conducted the PAR to meet the needs of the project,

while I conducted and directed the meta-evaluation to meet the requirements of this doctoral thesis.

Participatory action research (PAR) pursues both action and research outcomes. Its purpose is to

bring about planned change in such a way that there is also a contribution to the understanding of

the system that is being changed (Cunningham, 1993). In this thesis the PAR component involved a

12-month trial of a specific model of evaluation. The trial was conducted with the staff and

committee members of the Target 10 Dairy Extension Project in Victoria. These project

stakeholders collected and interpreted the data, which formed part of the evaluation, while I

5

initiated the trial and acted as facilitator of the process. Staff and committee members modified the

MSC process as it was implemented to meet regional needs and local context. I collected data

concerning the effectiveness of the model during the implementation that contributed both to

immediate modifications and to the meta-evaluation.

Meta-evaluation is the evaluation of an evaluation (Scriven, 1991). Theory-guided meta-evaluation

is an approach to ‘evaluating evaluation’ which involves constructing causal models of how the

evaluation is understood to improve the project (Rogers, 1996). The meta-evaluation was inductive

and iterative in design and involved two interconnected parts: inductive theory building to

understand how the evaluation process was working; and data collection to determine the extent to

which the process was effective, and in which situations and why. This data consisted of field notes,

meeting notes, semi-structured interviews and an on-line questionnaire administered to project staff.

1.5 Road Map for Thesis

The research for this thesis was conducted in an iterative manner; as one set of data was collected

and interpreted, this informed the design of the subsequent phase of data collection. As a

consequence of the iterative design, several methods were used at different stages of the research

cycle. Figure 1 represents a ‘road map’ of the thesis in which the colour coding represents a

distinction between chapters that focus on the macro national level (purple), the micro project level

(blue), or both levels (pink).

6

Figure 1 ‘Road Map’ of layout of thesis and research

Evaluation of the MSC evaluation of the Target 10 extension project (Chapter 8)

What was the impact of the MSC evaluation on the project? How well did it meet the evaluation needs of the project?

Evaluation of the MSC approach for its potential to meet needs of agricultural extension projects nationally (Chapter 9)

How good was it? How well did it address the challenges identified?

Description of modification and implementation of approach across an extension project (Chapter 7)

What happened when the MSC approach was

implemented?

Method (Chapter 6)

How can the evaluation approach be tested?

Develop a normative framework for what entails good evaluation based on global literature (Chapter 2)

What is good program evaluation?

Review current evaluation practice at a national level to identify gaps (Chapter 4)

What extension evaluation is occurring nationally?

Determine the challenges particular to context of Australian Agricultural Extension (Chapter 3)

What are the evaluation needs of extension organisations at the national level?

AIM: Develop a model of evaluation that has potential to help address the current challenges associated with evaluating the new genre of agricultural extension projects in Australia

Choose a novel approach to evaluation - consider needs and gaps at the project and national level (Chapter 5)

Which approach to test?

Conclusions (Chapter 10)

What are the implications for the evaluation of agricultural extension programs in Australia?

7

1.6 Overview of Chapters

Chapter 2 addresses the question: What is good evaluation? I argue that no single normative model

for evaluation can encompass the diverse and multi-faceted range of tasks required of evaluation in

large projects. After presenting an overview of the different facets of program evaluation, a series of

premises for ‘good’ evaluation are presented to guide the development of a new model of

evaluation for agricultural extension. A key premise is that evaluation should lead to programs that

better meet needs. A further premise is that the best way of conducting an evaluation depends on the

purpose of the evaluation and the program context. From these premises, a series of meta-evaluation

questions are developed that will be used in the subsequent chapters to guide the development of a

new evaluation model for extension.

Chapter 3 addresses the question: What are the evaluation needs in this particular context? The

current context and challenges associated with the evaluation of agricultural extension projects in

Australia are examined. The chapter begins with a brief look at changes that are occurring at a

global level and how these changes are reflected in the practice of agricultural extension. A

description is provided of the development of extension models from the transfer of technology

paradigm to new approaches influenced by the farmer-first paradigm. An emphasis is placed on the

move towards more participatory, group-based and process-oriented forms of extension. I suggest

that new models of extension require new models of evaluation that can deal with diversity in

implementation, participation of stakeholders within the evaluation study and an ability to provide

accountability to projects with loosely defined outcomes at the onset. Following this, changes

occurring in organisations that deliver extension are examined. An overview is provided of the

impact of recent changes in public sector management within the Agriculture Division, the

Victorian Department of Natural Resources and Environment. An attempt is made to unravel the

various reforms and to describe the consequent challenges posed for the practice of project

evaluation in agricultural extension.

Chapter 4 addresses the question: What evaluation is already occurring? The findings of a review

of evaluation in agricultural extension in Australia are presented in which 50 evaluation documents

were analysed using Owen’s (1993) meta-model that is based on distinct forms or purposes of

evaluation. Nine case studies are presented that illustrate the range of forms and approaches in

practice. Where possible, the evaluation studies are examined against the meta-evaluation criteria

laid out in Chapter 2. I found that there is very little monitoring of projects, and a tendency to stick

8

to a handful of familiar methods and approaches. Most of the studies were limited to a summative

assessment of project impact, and there was a pre-dominance of ‘black-box’ approaches. I argue

that while there is a range of forms and approaches to evaluation in practice, the suite of approaches

that are currently used is inadequate to meet all challenges and demands for extension evaluation in

Australia. In particular, no evaluation study is found that facilitated on-going communication

between multiple project stakeholders.

Chapter 5 addresses the question: Which evaluation model to test? Four contemporary models of

program evaluation are examined in terms of their ability to address both the macro needs of

extension projects nationally and the micro needs of the case study project across which it was to be

tested. The models examined are Patton’s (1997) ‘Utilization Focused Evaluation’, Pawson and

Tilley’s (1997) ‘Realistic Evaluation’, Guba and Lincoln’s (1989) ‘Fourth Generation Evaluation’,

and Davies’ (1996) ‘Most Significant Change’ model. All four models are found to have much to

offer extension evaluation at a national level. Davies’ (1996) Most Significant Change (MSC)

model is selected for application, primarily as it appeared to complement the existing evaluation

practice of the Target 10 Project and meet the evaluation needs of the project. The MSC model

employs novel methodology based on the collection and selection of stories of significant change. I

suggest that the use of stories could offer an effective way of creating dialogue and ‘sensemaking’

between the multiple stakeholders of large extension projects, which was one of the evaluation

needs of the case study project.

Chapter 6 addresses the question How to test the evaluation model? The research approach and

methodology employed for the case study evaluation and meta-evaluation are described and

justified. The research approach is based on a combination of Participatory Action Research and

non-participatory theory testing.

Chapter 7 addresses the question: What happened when the MSC model was implemented? The case

study is presented, comprising a description of the Target 10 Project and the implementation of a

12-month trial of the MSC model. The problems and issues that arose while implementing the

model are discussed. The output of the process included 134 stories, and feedback from the review

of these stories during 15 meetings of project stakeholders.

Chapter 8 addresses the questions: What impact did the MSC model have on the Target 10 Project

and how well does it meet the evaluation needs? To answer these questions, the findings of a

9

theory-guided meta-evaluation of the MSC evaluation of the Target 10 Extension Project are

offered. Firstly, a model is presented of how the process was thought to bring about improvements

to the project. From this model nine conjectured outcomes are identified which included: increased

participation, increased knowledge, increased staff morale, and operational and strategic changes to

extension practice. Secondly, empirical evidence is used to test whether the outcomes stipulated in

the model had been achieved, in what contexts and why. Significant regional differences are found

in the extent to which respondents perceived the outcomes to have been achieved. I suggest that

these regional differences were associated, amongst other things, with the structure and nature of

participation in the meetings at which the stories were reviewed. Despite regional differences in

magnitude of success, the MSC process was able to meet many of the current evaluation needs of

the Target 10 stakeholders. The MSC process also appears to complement the project’s existing

evaluation practice.

Chapter 9 addresses the question: How good is the MSC model, and how well can it address the

challenges facing large extension projects? To answer this question the MSC model is examined

against the key meta-evaluation questions developed in Chapter 2. I found that the MSC model goes

a long way to meeting the evaluation needs at the national level. In terms of leading to projects

being better able to meet needs, this will be somewhat dependent on the individual project context. I

suggest that this model may be particularly appropriate for large extension projects with diversity in

outcomes, with a high degree of farmer participation in project planning and decision-making. On

further examination against the meta-evaluation questions, the model satisfies some of the premises

for good evaluation, but also has inherent biases and weaknesses. This emphasises the need to

combine the MSC model with complementary evaluation approaches, to help meet all evaluation

needs and also to offset the bias.

Chapter 10 addresses the question: What are the implications for the evaluation of agricultural

extension programs in Australia? The chapter provides a conclusion to the previous chapter and

concludes the overall thesis.

10

CHAPTER 2

A CONCEPTUAL FRAMEWORK FOR GOOD PROGRAM

EVALUATION

In this chapter I argue that no single normative model for program evaluation can encompass

the diverse and multi-faceted range of tasks that evaluation involves. An overview of the

different facets of program evaluation, followed by a series of premises for ‘good’ evaluation

are presented to guide the selection and development of a new model of evaluation for

agricultural extension. A key premise is that evaluation should lead to programs that better

meet stakeholder needs. A further premise is that the best way of conducting an evaluation

depends largely on the evaluation questions and the program context. I suggest that a

bricolage approach to evaluation be employed: after analysis of the program context,

evaluation needs and existing practice, the bricoleur picks and chooses from various

evaluation models and tools and adds a new component to the bricolage. A further step

suggested is that the new evaluation tools themselves should be piloted, evaluated and

modified if necessary.

2.1 Introduction

The development of any new model of program evaluation should be guided by current theory and

practice. To select appropriate theories and practice from the large body of knowledge known as

‘program evaluation’, an appreciation of the current status and developments in this field must be

gained. It is also essential to establish some conception of what ‘good’ evaluation entails.

The aim of this chapter is provide a conceptual framework for program evaluation that will guide

this thesis. In simple terms this chapter addresses the question: What is good program evaluation?

This question is addressed by firstly determining what is implied by ‘program evaluation’, then by

examining different perspectives on ‘good evaluation’. A major source of theory and ideas are the

11

journals and texts entitled ‘program evaluation’ (that mainly relate to the evaluation of education

and health related programs). A sub-set of literature that offers practical suggestions and novel

approaches, is evaluation associated with development studies. This includes evaluation of projects

that are delivered in low-income countries, but often funded by organisations from other countries.

In development studies, program evaluation is commonly referred to as either ‘impact evaluation’

or ‘monitoring and evaluation’.

I begin the chapter by presenting definitions of evaluation and describing the literature of program

evaluation. I trace the historical beginnings of program evaluation through to the state of great

diversity that characterises program evaluation today. In an attempt to make sense of this

burgeoning field, I offer a meta-model that accounts for six broad approaches to evaluation.

Through an examination of these different approaches, I highlight some of the fundamental issues

surrounding the discourse in the field. Following this, I present a theoretical framework for

characterising ‘good’ evaluation, in the form of a series of meta-evaluation questions. I use this

framework to review examples of evaluation practice in Australia (Chapter 4), to choose an

approach to apply (Chapter 5), and finally to evaluate the case study evaluation (Chapter 7).

2.2 What is Program Evaluation?

Program evaluation today is characterised by great diversity: from large-scale, long-term,

international comparative designs costing millions of dollars, to small-scale short evaluations of a

single component in a local agency. Worthen et al., (1997) suggested that over 60 different models

for the conduct of evaluation have been developed. Each model has built-in assumptions.

Evaluation literature is vast and fragmented and appears to be aimed mostly at evaluation theorists

rather than practitioners. There are at least 17 international journals on evaluation. The discipline

has its own societies, books, awards, conferences and standards.

There is no single recognised definition of program evaluation. Historically, the definitions of

evaluation and program evaluation have changed constantly, reflecting the evolving trends in

thinking and practice. Early evaluation texts define evaluation narrowly as the application of social

science methods to measure goal attainment or the systematic investigation of the merit or worth of

an object. More recent definitions include the concept of using the evaluation findings for program

improvement, development, and decision making. For example, Owen (1993:3) defined it as ‘the

process of providing information designed to assist decision-making about the object being

12

evaluated’. The matter of defining evaluation is of considerable importance, as different evaluation

approaches rest on different definitions of evaluation. Before turning to an analysis of contending

issues and definitions for program evaluation, it is worth clarifying that in general program

evaluation does not include the:

• evaluation of student performance, course or curriculum evaluation as in educational evaluation

• personnel or staff evaluation as in Human Resource Management and personnel appraisal

• evaluation of computer programs or software (Sharp and Lindsay, 1992:6).

This thesis refers to both ‘program evaluation’ and ‘project evaluation’. The terms ‘program’ and

‘project’ are used to differentiate between different levels of organisational activity. In some

organisations, a program is considered to be a large-scale intervention comprised of a series of

smaller scale projects, yet in other organisations the opposite meaning is adopted. These terms have

context-specific meaning within organisations. Similarly, the terms ‘program evaluation’ and

‘project evaluation’ refer to organisationally-specific levels of evaluation practice. In international

evaluation literature there is a tendency to refer to ‘program evaluation’, yet in agricultural

extension it is more usual to refer to ‘project evaluation’. The two terms will be used

interchangeably throughout this thesis, and should be regarded as synonymous.

The various terms used in evaluation (models, approaches, method, and tools) can also be

confusing, so an attempt is made here to explain the relationships between various terms and how

they are differentiated in this thesis. Later in the thesis, the terms ‘meta-model’ and ‘forms’ of

evaluation will also be used, and are included in this list.

• Models of evaluation are based on a normative understanding of how evaluation should be

conducted. These are usually associated with known authors who promote their particular

model of evaluation.

• Approaches of evaluation: a collection of similar models or methods of evaluation, or ways of

going about evaluation.

• Methods of evaluation: methods of data collection, retrieval and creation.

• Tools of evaluation: a loose term signifying anything that helps to conduct an evaluation study.

This includes evaluation models, methods of data collection, retrieval and creation, even

methods of data analysis.

• Forms of evaluation: Owen (1993) differentiated between five different types of evaluation,

13

referred to as ‘forms’, based on the purpose of the evaluation study. ‘Forms’ are used in

Chapter 4 to distinguish different evaluation studies in agricultural extension practice.

• Meta-model of evaluation: a framework for describing or categorising various different kinds

of evaluation. The categories of a meta-model may be differentiated by model, approach, forms

or methods.

The historical development of program evaluation

Many popular texts on evaluation begin with an overview of the history of evaluation. Some

describe the history of evaluation in terms of the predominant paradigms (Pawson and Tilley,

1997), others in terms of the predominant actors in evaluation (Shadish et al., 1995). Guba and

Lincoln (1989) describe four generations of evaluation: description, measurement, judgement; and a

‘fourth generation’ model of their own (Fourth Generation Evaluation). The following summary

incorporates some of the prominent features in the history of evaluation, according to Madaus et al.

(1983), Shadish et al. (1995), Guba and Lincoln (1989), Worthen et al. (1997) and Patton (1997).

Wherever possible, an Australian perspective is also provided, based largely on the work of Sharp

and Lindsay (1992).

Evaluation has a long history originating from many diverse disciplinary fields – including

agricultural extension. In many western countries, including Australia, trends in program evaluation

appear to be linked to changes in public policy. The historical development of program evaluation

has not been a smooth one. There has been a tendency towards conflict and short memory, with

regards to the shortfalls of one approach against another. The diversity of backgrounds of the

evaluator-theorists can perhaps also explain some of these conflicts. Despite much effort, it seems

that the quest for the ‘holy-grail model of evaluation’ to satisfy all audiences, from all program

contexts, was in vain.

The beginnings of program evaluation

Between 1930 and 1945, the work of Tyler came to have an enormous effect upon the field of

evaluation. Tyler (1967) coined the term ‘educational evaluation’, which meant assessing the extent

to which valued objectives had been achieved in an institutional program. Evaluation was

conceptualised by Tyler as a comparison of intended outcome with actual outcomes. The currently

popular outcomes-oriented evaluation is clearly not a new concept for program evaluation and this

highlights the circular nature of the debates within this field.

14

Most texts agree that modern program evaluation evolved primarily in the USA and was considered

a ‘semi-professional’ discipline by the 1960s. Although the root of evaluation development lies in

the US, in the 1960s, evaluation began to surface in Australia and later in Europe. The Baume

Report (Baume, 1977) traced the history of health and welfare evaluation in Australia from

Federation to the Whitlam Government (1972), and stated ‘there was almost no formal evaluation

during this time’ (cited by Sharp and Lindsay, 1992, Senate Standing Committee on Social Welfare,

1979a: 19). Despite this claim, there were some who can be identified as active practitioners and

theorists in the 1960s and earlier. They participated in the predominantly educational fields of

agricultural extension and Technical and Further Education (Sharp and Lindsay, 1992). The first

national evaluation conference in Australia was held in 1982.

Influences of government policy

Much of the history of program evaluation appears to be linked with government policy. Shadish et

al. (1995) suggest that in the USA an increased interest in evaluation in the period between 1946-

1960 can be largely attributed to the rapid economic growth after World War II, and in the

interventionist role many Western governments took on social policy during the 1960s. In the USA

during this period, educational services and social programs were expanded in efforts to address

rural poverty and despair in the inner cities (Madaus et al., 1983). The increase in public spending

on programs led to an increase in evaluation activity to determine whether these programs were

working.

Around 1960 the US government invested large sums of money towards programs in education,

income maintenance, housing, and health. These programs are collectively termed ‘Great Society’

programs. Expenditure on these programs was massive and constituted an 18-fold increase in US

public spending between 1950 and 1979 (Bell, 1983). This vast expenditure raised issues that

resulted in evaluation being promoted as a necessary component of social programs. Issues of

accountability for distribution of funds and concerns that program funds were being spent in ways

that caused undesirable results led to an increased demand for evaluation for accountability.

While large-scale programs with an evaluation component became commonplace in the US, this

trend also occurred in Europe and Australia. By the late 1960s in the United States and

internationally, evaluation research had become a ‘growth industry’ (Wall Street Journal, cited by

Rossi et al., 1979). In Australia, under the Whitlam Government of the 1970s, expansion of

15

government activities was seen in many fields. Many commentators regard the Whitlam era as

significant for its emphasis on social welfare programs, and with it among the first attempts to

address the policy and practice of program evaluation across many of the public sector services in

Australia (Sharp and Lindsay, 1992).

A multi-disciplinary enterprise

Along with Tyler’s educational model, the theory and practice of program evaluation had early

roots in diverse disciplines, including social psychology (Lewin, 1948), sociology (Lazerfield and

Rosenburg, 1955) and agricultural extension (Fisher, 1956; Rogers, 1962). Nowadays, evaluation

seems to be practised in almost every field of work, and yet not all of these get a mention in the

main texts on program evaluation. The literature of program evaluation is so vast that subsets exist,

which operate quite independently of one another. In particular, evaluation has long been practised

in the field of development studies: that is projects that are delivered in low-income countries, but

funded by donor organisations. Development projects were evaluated from as early as 1970 and

today there is still a strong tradition of evaluation (Jiggins, 1995).

In Australia, in the sectors of education, health and welfare, program evaluation was not made

mandatory or endorsed as government policy until the late 1980s. However, Sharp and Lindsay

(1992) point out that in the fields of agricultural extension and technical education, program

evaluation became the prescribed, widely implemented policy of the government agencies

concerned as early as 1968. Sharp contends that both in Australia and in US in this period,

agricultural extension came to have a great influence on evaluation practice:

Like the role of agricultural research on the development of statistical methods (Fisher, 1956) and

research on the diffusion of innovation (Rogers, 1962), the research on agricultural extension

provided opportunities which had a formative effect on the early approaches to program evaluation.

Some of the current practitioners of program evaluation in the human services gained their

experience through agricultural extension program evaluation. For example, Michael Quinn Patton

(former President of the American Evaluation Association) was involved for many years in extension

services through the Institute of Agriculture, Forestry and Home Economics at the University of

Minnesota (Sharp and Lindsay, 1992:11).

In 1947, some major projects in Australian agricultural extension were evaluated as part of their

planning and funding process. By 1962, South Australia had introduced a compulsory training

program in program planning and evaluation for agricultural extension officers – and a similar

16

compulsory scheme was introduced in New South Wales by 1968 (Sharp and Lindsay, 1992). In the

Murrumbidgee Irrigation Area (MIA), an agricultural extension project was conducted with a

deliberate research and evaluation strategy by the CSIRO, state and local authorities (Farquhar,

1961). This evaluation was conducted by trained evaluators and it was reported that ‘many lessons

were learned from the careful evaluation of the MIA extension experiments…probably the most

significant was the lesson in agricultural organisation’ (Farquhar 1961:210).

Many texts, many societies, many differences of opinion

Before 1970, limited published literature about program evaluation existed; many evaluations were

carried out by untrained personnel, others by research methodologists who tried to fit their methods

to program evaluation (Guba and Lincoln, 1981). Responding to the need for more sophisticated

evaluation models, many new books and papers were published by 1970, especially in the USA

(Cronbach, 1963; Scriven, 1967). Around 1973, a number of journals, including Evaluation Review

and New Directions in Program Evaluation were launched. In the USA in the 1970s, two

professional evaluation societies were formed: the ‘Evaluation Research Society’ and the

‘Evaluation Network’, later combining to form the ‘American Evaluation Association’. By 1986 the

International ‘Australasian Evaluation Society’ and the ‘Canadian Evaluation Society’ had formed,

and by 1995 new professional evaluation societies had been established to represent Central

America, Europe and Great Britain.

Together these new publications and fora for debate stimulated a number of new evaluation models

in response to the different situations and needs of specific types of evaluation. This body of

information revealed sharp differences in philosophical and methodological preferences, but ‘it also

underscored a fact about which there was much agreement: evaluation is a multidimensional

technical and political enterprise that requires both new conceptualisation and new insights into

when and how existing methods from other fields might be used appropriately’ (Worthen et al.,

1997:39).

Another issue that is highlighted by an historical perspective is the heavy influence of work from

the USA on evaluation theory. Many texts used in Australia are still produced in the USA; the

extent to which these texts are wholly relevant to the Australian context is debatable. The

development of program evaluation in Australia differed somewhat from that in the US, in that

internal evaluation was always more prevalent than in the US. This can perhaps be explained by the

attitude in the US to evaluation as an academic discipline and the existence of the numerous

17

evaluation higher degree courses there. In the US, evaluation tends to be seen as something that is

done by trained external evaluators, qualified with a PhD, whereas in Australia there tends to be a

greater focus on internal evaluation (Winston, J. pers. comm., 1997). There is certainly an emphasis

on internal evaluation in the field of agricultural extension in Australia (Dart et. al., 1998).

Diversity of evaluation models and the paradigm wars

Shadish and Epstein (1987) suggest that, as evaluation matured, its theory took on a special

character that resulted from the interplay among problems encountered by practitioners from

different academic disciplines. In a later text, Shadish et. al. (1995) go on to suggest that:

Evaluation may be the broadest methodological speciality…to inform evaluators about choosing

methods, it needs to discuss philosophy of science, public policy, value theory and theory of use

(Shadish et al., 1995: 31).

In the multi-faceted field of program evaluation, several paradigm wars raged throughout the

second half of the 20th century. Early differences concerned the focus on internal validity

(Campbell, 1969) against a consideration of external validity of evaluation findings (Cronbach,

1963). Another argument concerned the exclusive emphasis on pre-determined outcomes, against a

need to understand the quality of the implementation and the causal process that mediated program

impacts (Sechrest et al., 1979). However, the most cited debate has been between the constructivist

stance (e.g., Guba and Lincoln, 1989) and post-positivist stance (e.g., Campbell, 1969), which

correspond to qualitative and quantitative methods respectively. This particular conflict became

increasingly heated from 1970 until the 1990s. Datta (1994) notes that the ‘dialogues’ of three

successive presidents of the American Evaluation Association (Fetterman, 1992; Lincoln 1991;

Sechrest, 1991) were stringent in their defence of their methodological positions and in their attack

on the position of the other side. Datta suggested that this paradigm war became increasingly

unproductive during the 1980s and 1990s.

There have been attempts by several scholars to make peace in this paradigm war, with the

increased use of multiple or mixed methods (Cook and Campbell, 1979; Cronbach et al., 1980;

Datta, 1997).

18

2.3 What is ‘Good’ Program Evaluation?

As described, evaluation is used in many program contexts, in many countries and across many

different disciplines. It is also used for many different purposes, and the range of things that are

included under the umbrella of program evaluation is considerable. Even within one project there

may be several evaluation initiatives under way. For this reason, in ‘good’ evaluation the choice of

evaluation approach needs to be context-specific and take into consideration the purposes for which

the study is being undertaken. This view is endorsed by many evaluators including Worthen et al.

(1997) and Owen (1993).

Notwithstanding such recognition, several authors have advanced normative models for what they

believe entails good evaluation. Predictably, there are considerable differences between these

models. Despite the differences, an examination of these models can offer many insights into what

entails good evaluation. Therefore, six different approaches to evaluation were examined in this

study and from these a series of premises for good evaluation were developed. I propose a

classification of evaluation consisting of the following six normative approaches, each of which will

be discussed in turn:

i. Experimental approaches

ii. Testing-objectives approaches

iii. Decision-management approaches

iv. Judgemental approaches

v. Pluralist-intuitionist approaches

vi. Theory-driven approaches

These six approaches are an amalgam of the meta-models of Smith (1994), Stake (1973), House,

(1978), Stufflebeam and Webster, (1981) and Worthen et al. (1997). In reality this classification,

and the ones it is based on, are over-simplistic because of the complexity of the models they attempt

to classify. Wadsworth (1991: 62) points out that ‘One would really need a three-dimensional map

on which to try to plot a full picture and even then some of the techniques would have to be moved

back and forth between one category and another’. However, the meta-model proposed does allow

an examination of the range of things that theorists contend are important for ‘good’ evaluation.

19

The theories behind both of the first two approaches in the meta-model (experimental approaches

and testing-objectives) have been partly superseded by more modern approaches. However, they are

included in the meta-model because they are still seen in practice and because critique of these

approaches paved the way for the more contemporary approaches; theory-driven, judgemental,

decision-management and pluralist-intuitionist.

2.4 Six Normative Approaches to Evaluation

i. Experimental approaches

Experimentalism, often referred to as the ‘classical paradigm’ in program evaluation, is

characterised by the work of Popper (1959), Campbell, (1991), and Cook (1966). Experimental

approaches construe evaluation as a knowledge generation research task. Therefore good evaluation

creates knowledge and theory about a program situation that can be ratified by empirical data.

Campbell focused much of this energy on the notion of minimising the ‘threats to internal validity’,

and developing the most plausible explanations for the results of experiments.

Experimental approaches are based on a Humean theory of causation. The basic task is to

hypothesise or demonstrate the constant conjunction whereby action ‘X’ produces outcome ‘Y’.

Advocates of such approaches promote the use of experimental and strong quasi-experimental

designs that provide the strongest causal inference. Experimental designs in evaluation in principle

follow the same rules as agronomic trials: the ‘treatments’ are randomly assigned (in this case to

program participants) to either a ‘treatment’ group or to a control group. Experimental approaches

may also have pre- and post-test designs, so that changes can be monitored before and after the

treatment period.

Experimental designs offer a logical approach for determining whether certain program variables

affect program outcomes. While experimental approaches can be criticised on several fronts, there

is little doubt that experimental evaluation paved the way for important insights into evaluation

theory, for example, the work of Campbell (1991, 1969) and Cook (Cook and Campbell, 1979) who

focused on the ‘internal threats to validity’ associated with experimental design.

However, for several reasons, experimental designs did not always prove their value, and many

20

expensive designs yielded inconclusive findings. Returning to the analogy of agronomic trials, the

secret to a good experiment is to ensure that the plots are randomly allocated and that all other

variables are controlled. Even with plants, this can produce results that are difficult to apply to the

reality of the farmers’ fields, as the controlled conditions of the research station do not correspond

to real life conditions. It can be even more difficult to control variables amongst human participants

and it is often unethical to subject one group of people to a treatment, and deny it to others. Even

where large trials have been conducted, the results have been disappointing. Variables between the

responses of different people within the same treatment group are seen as ‘noise’ in the experiment.

Therefore, the results of an experimental design only provide information as to whether the program

‘worked’ for the entire group of people or not. It provides no clues as to why the program may work

for some individuals and not for others.

Indeed, for practical purposes, experimental designs often exclude many of the contextual factors

that influence cause-and-effect relationships. It can be argued that these contextual factors are the

very thing in which evaluators should be most interested. Theory-driven approaches to evaluation

for example, reject a key notion inherent in the logic of experimental evaluation in that it cannot

fully take into account either the key mechanisms linking programs with outcomes or the richness

of heterogeneous contexts (Feinstein, 1998). Indeed, theory-driven evaluators such as Pawson and

Tilley (1997) attack experimental evaluation for yielding very little in terms of learning about

programs. Their reasoning is:

By its very logic, experimental evaluation either ignores underlying process, or treats them

incorrectly as inputs, outputs or confounding variables, or deals with them in a post hoc and thus

arbitrary fashion (Pawson and Tilley, 1997:54).

Despite their limitations, experimental designs are still used and valued in many circles – especially

in the United States.

ii. Testing-objectives approaches

Evaluation approaches in the testing-objectives category are focused on determining whether the

stated goals or objectives of a program have been achieved. Tyler (1967) was among the first to

develop this approach which he referred to as ‘educational evaluation’. Good evaluation under this

model depends on being able to accurately determine the extent to which stipulated objectives have

been reached. Tyler defined evaluation as the process of determining the extent to which the

21

objectives of a program are attained (Worthen 1997:82).

Guba and Lincoln (1989) relate, how in 1933, Tyler was engaged to carry out an eight-year study of

an alternative school curriculum. The idea was to collect information about the extent of pupil

achievement against defined objectives, to guide revisions to the curriculum. The results of each

trial were not available until after the trial was complete. This process was iterated over successive

course offerings until the curriculum was found to produce an appropriate level of achievement. In a

sense, this was formative evaluation and the evaluators’ role was that of describing the attainment

of objectives over time. A point to note about this approach, however, was that the results of the

evaluation were not made (publicly) available until after the program was complete. This implied

that the program could not be modified during implementation. This approach to evaluation is still

used in extension evaluation nowadays.

Objectives-testing approaches differ from the previously popular model of ‘comparative

experiment’ in that they do not involve the expensive and disruptive comparisons between

experimental treatments and the control. Since Tyler’s approach calls for the measurement of

behaviourally defined objectives, it concentrates on learning outcomes instead of organisational and

teaching inputs (Madaus et al., 1983). Educational evaluation that predated objectives-testing

approaches tended to focus on measuring the performance of pupils, rather than examining the

curriculum or program itself.

Careful articulation of the objectives of a program – an essential part of an objectives-testing

approach – can have positive ramifications for both program planning and delivery. Very few

evaluations are conducted without some consideration of the extent to which objectives have been

achieved.

However, critics of objectives-testing approaches found it unsatisfactory to be unable to make

corrections to the program until it was complete. In other cases, practitioners were unwilling to

stipulate pre-determined ‘outcomes’, as they were unsure at the onset of the programs about the

appropriate outcomes. Both of these criticisms can still be heard today of evaluations that are

strongly outcome (or objectives) focused. Another important criticism of this model was levelled by

Stake (1967), who pointed out the neglect of judgement in the objectives-testing model. He

suggested that evaluation requires standards against which judgements can be made and that the

inclusion of standards must be value-laden. Scriven (1967) also pointed out that the objectives

22

themselves should also be treated as problematic and subject to scrutiny. His argument was that

testing the extent to which the goals of a program have been achieved does not determine the worth

of that program for society in general; it does not assess the goals themselves.

iii. Decision-management approaches

Decision-management approaches aim to serve decision-makers’ needs in managing programs.

Evaluation models fitting into this category are Patton’s Utilization Focused Evaluation (1997), and

Stufflebeam’s CIPP model (context, input, process, and product) (Stufflebeam and Webster, 1981),

that relies on a modified systems analysis approach. Good evaluation, under this approach, would

produce findings that are used to bring about effective decision-making for the organisation. This

type of evaluation is largely judged by how well it has served those commissioning the evaluation.

For example, (Caron, 1993: 62) states that evaluation is ‘carried out for the organisation. It is a

function of management’. Therefore, in evaluation of this genre the aim is to serve the needs of

program managers rather than of the program’s clients or the wider public.

Patton, who is probably the most well cited advocate of decision-management approaches, defines

program evaluation as:

…the systematic collection of information about the activities, characteristics, and outcomes of the

programs to make judgments about the program, improve the program effectiveness, and/or inform

decisions about future programming. Utilization focused program evaluation (as opposed to program

evaluation in general) is evaluation done for and with specific, intended primary users for specific,

intended uses (Patton 1997: 23).

According to Patton (2000:1) Utilization Focused Evaluation is an ‘approach to making evaluations

useful, practical, accurate, systematic, and ethical’. This involves matching the evaluation approach

and the design to the information and decision needs of primary intended users, taking into account

‘other stakeholders, political factors, organisational constraints, project/program history, available

resources, and cultural factors of a specific evaluation context’ (ibid: 1). He suggests that this

allows for ‘situationally responsive’ evaluations.

While decision-management approaches offer sensible suggestions for increasing the likelihood of

the evaluation findings being used, they have been criticised for the apparent ‘cosy’ relationship

between the evaluator and the project management. For example Pawson and Tilley (1997) caution

that these pragmatic approaches ‘suggest a Rothschildian vision with research skills for hire’ – the

23

Rothschild principle being:

the customer says what he wants, the contractor does it if he can, and the customer pays (Department

of Health, 1993; cited by Pawson and Tilley 1997: 14).

Thus, decision-management models are challenged on their ability (or inability) to present

unpalatable information to the management. Some decision-management models can also be

questioned on their assumption that evaluation serves the needs of program managers and staff

rather than program clients or the greater public.

iv. Judgement approaches

Judgement approaches involve the professional judgement of experts. This approach includes

Scriven’s (1976) ‘Goal-free’ evaluation and Eisner’s (1985) Connoisseurial model of evaluation.

Here, evaluation is seen as a determination of the merit or worth of a program – and an evaluation

of this type would be judged on the basis of the accuracy and lack of bias in conducting a

judgement of worth or merit of the program. Definitions of evaluation for this are characterised by

the words of Scriven:

It’s extremely important not to over simplify the logic of evaluation by defining evaluation as, for

example, the provision of information to decision-makers. Evaluation is what it is, the determination

of merit or worth, and what it is used for is another matter (Scriven, 1980: 8).

Bad is bad, and good is good, and it is the job of evaluators to decide which is which. (Scriven, 1986:

19)

An extreme model from this genre is Scriven’s ‘Goal-free Evaluation’. ‘Goal-free evaluators begin

evaluations totally blind’ (Scriven, 1976: 137) to stated goals. Evaluators have to discover what

effect the program has and match their effects against the needs of those who they affect (Scriven,

1976: 137). The ‘Goal-free’ evaluator avoids contact with program staff, who may bias the

conceptualisation of the evaluation questions. The evaluator is required to judge the merit of the

program in terms of empirical evidence of the effects of the program intervention.

In Goal-free Evaluation, Scriven attempts to address the problem of over-emphasis on program

outcomes (or objectives), stating that they should be totally ignored. The evaluator’s job, according

to Scriven, is to locate any and all program effects, intended or not, that might help solve social

24

programs. His argument is that testing the extent to which the goals of a program have been

achieved does not determine the worth of that program for society in general; it does not assess the

goals themselves. Goals are a poor source of such effects, and are ‘often vaguely worded to muster

political support and rarely reflect side effects that are difficult to predict’ (Shadish et al., 1995:80).

Goal-free Evaluation does serve to remind evaluators of three very important points. Firstly, in

addition to examining stipulated outcomes, evaluation should look beyond the goals of the program

itself, ie., it should consider unexpected outcomes. Secondly, in some situations it may also be

necessary to question the value and logic of the outcomes themselves. A third contribution of Goal-

free Evaluation is the concept that good evaluation involves some judgement of merit or worth of

the program’s impact.

However, Shadish et al. (1995: 114) suggest that ‘some evaluators have difficulty accepting the

notion that they can, much less should, evaluate a program without knowing its goals’. While most

evaluators have heard of Goal-free Evaluation, they may not see it as central to their thinking about

evaluation, and they still use goals as the most common source of dependent variables (Shadish and

Epstein, 1987). Another criticism of the Goal-free Evaluation is that while it may be a very useful

theory, it is not necessarily a practical model. Indeed, few cases of Goal-free Evaluation have been

documented.

Critics of Goal-free Evaluation consider the term ‘goal-free’ evaluation to be a misnomer. The

evaluator does not get rid of all goals, but replaces the goals of the project staff with more global

goals based on societal needs and basic standards of morality (Alkin, 1972 cited by Patton

1997:183). Patton argues that Goal-free Evaluation:

…eliminates only one group from the game, local project staff. He (Scriven) directs data in only one

clear direction – away from the stated concerns of the people who run the program. He addresses an

external audience such as legislative funders. But, in as much as these audiences are ill defined and

lack organisation, I am unconvinced that the standards he applies are none other than his very own

preferences about what program effects are appropriate and morally defensible. Goal-free Evaluation

carries the danger of substituting the evaluator’s goals for those of the project (Patton 1997:182).

25

v. Pluralist-intuitionist approaches

Pluralist-intuitionist approaches share a common commitment to value pluralism; that is to identify

and preserve multiple value perspectives. This includes the models of Stake (1967), Guba and

Lincoln, (1989) and Wadsworth, (1991) and the participatory monitoring and evaluation approaches

to evaluation that are seen in development projects (Estrella and Gaventa, 1998). All the approaches

in this category are highly client centred and present a subjectivist ethic and epistemology within a

liberal ideology (Smith, 1994). Good evaluation under this normative model involves creating

increasingly sophisticated and shared constructions of reality. For example, evaluators employing

Guba and Lincoln’s ‘Fourth Generation Evaluation’ (1989) judge evaluation by the extent to which

it succeeds in involving a range of participants in sharing their views. Guba and Lincoln, (1989) for

example, advocate that evaluators ought to strive for, not validity, but increasingly sophisticated

constructions of reality. Some versions of pluralist-intuitionist approaches also focus explicitly on

social justice, and liberation of program beneficiaries from oppression.

Pluralist-intuitionist approaches evolved as a reaction to the limitations of the logical positivist

approach to knowledge generation such as that adopted in experimental approaches. To overcome

the limitations of these approaches, a number of overlapping frameworks emerged that were based

on the tradition of anthropology. Pluralist-intuitionist approaches include interpretivism, relativism,

naturalistic inquiry, constructivism, and feminist inquiry. What they hold in common is the rejection

of the existence of a singular knowable reality. This ontological position has significant

ramifications for the field of evaluation; evaluation is concerned with ‘apprehending reality’, thus

questioning the nature of ‘reality’ is clearly going to affect evaluation practice at several levels. In

Chapter 5, two models of this genre are examined in more detail.

One family of approaches that fits into this genre is the Participatory Monitoring and Evaluation

approaches (PM&E), also known as Participatory Learning and Action approaches. These

approaches are used increasingly in development projects. This type of evaluation is not generally

referred to in the main texts on program evaluation, but is gaining increasing attention in the

international literature of agricultural extension (Estrella and Gaventa, 1998; Guijt and Gaventa,

1998). Like Fourth Generation Evaluation, PM&E approaches share a common commitment to

participation of the beneficiaries of a program. However, PM&E approaches tend to emphasise

action a little more strongly, and to be influenced by models of action research. Good evaluation

26

under PM&E approaches would be based on the worldview of the people it is aiming to serve, and

would encourage participants to take self-directed action to improve their own or collective social

conditions. PM&E approaches were influenced by the Brazilian activist Friere, (1972) and the work

of action researchers such as Lewin (1948) and Whyte (1991).

PM&E approaches were developed in reaction to dissatisfaction with the rapid, yet participatory

approaches named ‘participatory rural appraisal’ (PRA), and the limitations and dangers of reliance

of non-participatory forms of monitoring. PM&E approaches are largely qualitative, participatory

approaches with a focus on organisational learning and empowering the beneficiaries of the project.

Pluralist-intuitionist models reinforce the notion that in good evaluation, a range of different

perspectives should be included. The evaluator is reminded that programs are conducted in a

political environment, and the values of the different stakeholders must be brought to the surface,

not excluded or ignored. Pluralist-intuitionist approaches offer a way of appreciating and

incorporating the different ‘theories of action’ or ‘constructions’ of the project held by different

stakeholders. These approaches can be especially valuable in the sort of evaluation that involves a

broad range of stakeholders involved with disparate views. They are often highly useful in

participatory programs, and particularly when the program intervention emphasises empowerment.

Like other approaches, pluralist-intuitionist models are not appropriate to every evaluation context.

In some cases, evaluations of this type may not meet the client’s information needs. Two examples

illustrate this point. Firstly, the client of the evaluation may not entertain the view that there are

many possible versions of ‘reality’ but instead may request a definitive account of the ‘facts’ about

program success. Secondly, clients who require firm recommendations to be a part of the evaluation

output may be dissatisfied with this approach. Evaluators who conduct pluralist-intuitionist

evaluation generally avoid making recommendations, and the evaluator may refrain from offering

concrete ‘recommendations’, and see their role as more of a ‘facilitator’ than a ‘judge’.

Many of the pluralist-intuitionist approaches, such as Fourth Generation Evaluation, strive to gain

consensus between all stakeholders. This can be questioned on two fronts. Firstly, the achievement

of consensus is not always a realistic goal, especially when different stakeholder groups may have

strong political differences. Secondly, the achievement of consensus is not always a helpful goal,

especially if it is won at the expense of ‘less powerful’ or minority opinions being silenced in

favour of the majority opinion.

27

vi. Theory-guided approaches

Recently there has been a growing interest in theory-guided approaches to evaluation (Pawson and

Tilley, 1997; Chen, 1990; and House, 1991). These approaches involve the construction of

‘program theory models’ – casual models that elaborate how a program is intended to achieve

intended outcomes. Bickman, (1987: 5) defines program theory as ‘the construction of a plausible

and sensible model of how a program is supposed to work’. Chen explains that program theory has

both descriptive and prescriptive concerns and defines program theory as:

A specification of what must be done to achieve the desired goals, what other important impacts may

also be anticipated, and how these goals and impacts would be generated (Chen 1990:43).

Good theory-driven evaluation would, therefore, develop highly plausible theory concerning how a

program works, in what situations and why. This theory would be the best explanation of empirical

data, and any alternative contending explanations would have been refuted by the empirical data. In

Chapter 5, one model of this genre is examined in more detail: Pawson and Tilley’s (1997) Realistic

Evaluation.

The advantages of theory-driven evaluation are that it goes beyond establishing whether a program

works, allowing an understanding of how a program works. Wortman (1983: 20) comments that

‘program evaluation is a multi-disciplinary and unfortunately largely atheoretical activity’. Chen

(1990) suggests that this atheoretical view tends to result in a simple input/output (or black box)

type of evaluation, characterised by adherence to a step-by-step cookbook method of doing

evaluation. He argues that such simple evaluations may provide evidence as to whether a program

works or not, but fail to identify the underlying mechanisms that generate the treatment effects, and

hence to pinpoint the deficiencies of the program for future program improvement. Therefore,

theory-driven approaches offer an important contribution to good evaluation: that evaluation should

move beyond merely asking why a program works, but look specifically at what aspects of a

program work in which situations and why.

Critics of this approach, such as Scriven (1994), disagree with the notion that understanding the

underlying theory of a project is critical and state that program theory is not needed for evaluation –

as it is not needed to determine whether something works or not:

28

In program evaluation, we are concerned to establish the merit, worth, quality or value of

programs…We do not need to know how the programs work or why they fail to work, or even what

their components are (Scriven, 1994:75).

Scriven (1994) contends that understanding the mechanisms behind an intervention can be

advantageous, but the main necessity is to get the evaluation right, and ‘jeopardising that by

diversion of effort into the direction of explanation diagnosis and remediation is all too common’

(ibid: 75). Indeed, it seems that in some cases, depending on the nature of the evaluation study, an

investigation into the underlying mechanisms might be something of an overkill; while unpacking

the black box is generally considered useful to any evaluation, it may not be necessary for all, and

its relevance depends on the nature of the evaluation inquiry (Scriven, 1994).

Depending on the nature of the evaluation questions, a theory-driven approach may or may not be

appropriate. Firstly, as argued by Scriven (1994), there are situations when the cost-benefit ratio of

conducting an extensive theory-driven evaluation is not justifiable. Secondly, in highly participative

programs there may be instances when the nature of the evaluation questions rightfully places more

emphasis on answering specific stakeholder concerns than on investigating carefully crafted

analytical suppositions.

Leviton (1994) states that theory-driven approaches are most seriously deficient in assignment of

value to community-based programs. She suggests that when ‘experts’ frame questions for the

community they encounter understandable fury. This is noteworthy as many extension programs are

highly participatory and community based in orientation. In many evaluation studies in extension,

the evaluators would endorse the position that the clients’ and key stakeholders’ values are the ones

that should be used to assign value. Thus, in these cases, the use of a deductive-normative theory

for developing the key evaluation questions would be rejected in favour of developing key

questions through the achievement of some sort of consensus from the project stakeholders with

regard to what they believe should be measured in the evaluation.

Theory-driven evaluation does not focus overtly on how unintended or unanticipated outcomes are

to be investigated. Because of this there is a tendency to get drawn into the micro mechanisms that

have been postulated, neglecting to look at the data afresh for new emerging patterns or

mechanisms. Hamilton-Smith and Hopkins (1998) suggest for example, that in Realistic Evaluation,

no overt demand is made to include unintended consequences once plausible and rival explanations

are developed. Theory-driven models have also been criticised as they do not focus on how the

29

evaluation findings are to be transferred into decisions that will lead to improved programs.

Commonalities and differences between evaluation approaches

The examination of different normative models illustrates that there are many different concepts

concerning ‘what evaluation is supposed to do’. However, it can be said that ultimately all of the

evaluation approaches have one thing in common; they attempt to bring about improved programs –

that is, programs that better meet the needs that they were designed to address. In turn, these

improved programs are all intended, in one way or another, to ameliorate social, economic or

environmental problems. Figure 2 shows a program theory model of the different approaches to

program evaluation. While each of the approaches strive to bring about improved programs, they do

this by different means, and therefore have different intermediate outcomes. Other authors have

presented program theory models for program evaluation including Patton (1997), Rogers (1996)

and Shadish et al. (1995).

Patton (1997) suggests that evaluation should be focused on using different organising concepts. He

suggests that in addition to program goals, evaluation can be focused upon: future decisions, critical

issues or concerns, key questions and the multiple perspectives of different stakeholder groups.

Figure 2 summarises the central organising concepts in each of the six normative approaches to

evaluation, as well as the outcomes and intermediate outcomes for each.

30

Figure 2

Figure 2 Outcomes and intermediate outcomes of six different normative approaches to

evaluation

The central differences between the six normative approaches reviewed, and summarised in Figure

2, give rise to the following questions:

• Which organising concepts should an evaluation be built around – goals, unexpected outcomes,

concerns and issues, or intended use?

• To what extent should the evaluation include judgement by the evaluator? What does this

judgement consist of? If not the evaluator, who should be making the judgements?

• What efforts should be made to ensure that the evaluation findings are used?

a) An improved project that can better meet needs or b) improvements in the next project

Needs are met and problems are ameliorated

Normative approach

Intermediate outcomes

Intermediateoutcomes

Outcome

Greater outcome

Management decisional approaches

Intended users involved in design and conduct of the evaluation – so accept findings

Information to make better management decisions

Key questions

Information about what works and does not work

Experimental approaches

Some of these cause-effect relationships are either refuted or supported

Cause-effect relationships

Judgemental approaches

Judgement of process and outcomes

Definition of standards of merit and worth against which to judge the approach

Intended and unintended outcomes and client needs

Organising concepts

Pluralist-intuitivist approaches

Intended users involved in design and conduct of the evaluation – so come to consensus

Mutual learning and self-directed action

Stakeholders concerns

Theory guided approaches

A greater understanding of the internal processes of the program

A refined theory of how the project does work in different contexts

Underlying program mechanisms occurring in contexts

Objectives testing approaches

Understanding of the extent to which defined objectives were met

Measurable objectives

Revised statements of objectives

31

• Who should the evaluation serve – society, the evaluation client, or the users of the service?

• To what extent should the key stakeholders be involved in the conduct and design of the

evaluation?

• Who should decide on the key evaluation questions or performance indicators, ie: who decides

what is measured?

• To what extent should evaluation be guided by the underpinning theory of the program

intervention?

These questions are not easily answered, yet the answers for each specific evaluation study may

guide the would-be evaluator toward a particular model of evaluation and away from others. I

suggest that within one large extension project it may be necessary to adopt more than one of these

normative approaches, entirely or partially, to meet the evaluation demands of stakeholders and to

lead to improved projects. Because of this, I suggest that a pragmatic approach be adopted, whereby

not only methods, but choice of normative approaches used to guide an evaluation be considered in

the light of the purpose of the evaluation study. In other words, the above questions could all be

answered with the statement ‘it depends on what you are trying to do’. This view corresponds with

that of Kaplan (1964) who suggests that the emphasis must be on making the methodology fit the

needs of the society, its institutions and its citizens, rather than the reverse.

Therefore, I advocate a framework for picking and choosing between different evaluation models.

‘Picking and choosing’ between different evaluation models can signify one of two things: choosing

one evaluation model in particular to guide the evaluation or choosing bits of different models, to

develop a ‘tailor-made’ evaluation. Either of the two approaches can be used.

2.5 Bricolage and Mixed-Approach Designs

Theorists argue about the extent to which it is acceptable to mix models and research paradigms

within an evaluation study. The contending points of view are that:

a) Research based on different discourse (or paradigms) is ‘incommensurable,’ i.e., cannot be

brought together in a coherent interpretation and cannot be reconciled (i.e., Guba and Lincoln,

1989).

b) Research is always potentially biased; hence we need to uncover through analysis and

argument, the biases inherent in different approaches and design approaches that systematically

32

address these biases, by putting together packages of imperfect theories and methods with

counterbalance the biases (i.e., Cronbach et al., 1980).

This thesis is premised on the second argument; that a combination of different models and tools

can better off-set bias, at the same time as meeting a range of evaluation needs. In the case of the

new genre of large extension projects, mixed methods designs occur in two distinct ways. Firstly, it

may be that within the evaluation bricolage several distinct evaluation approaches will be required

to serve different evaluation audiences. For example in a large hypothetical extension project, the

evaluation bricolage might consist of:

• internally conducted evaluation of the ‘logic’ of the core programs using Bennett’s Hierarchy

and subsequent refinement of the valued outcomes for each core program

• internally conducted evaluation of each of the core programs against valued outcomes

• externally conducted economic impact analysis of the whole project

• internally conducted reporting against predetermined milestones

• internally conducted evaluation of staff satisfaction, for project learning.

Secondly, mixed method studies may be used within one component of an evaluation bricolage.

Different methods may be used for different stages in the process of answering the same question,

or to validate the findings. For example, in the above scenario, the evaluation of achievement of

program outcomes could have been conducted by using a mix of focus group technique and

questionnaire.

In summary, the framework advocated in this thesis can be described as ‘bricolage’; a term that has

been used to describe qualitative research (Denzin and Lincoln, 1998) and cultural studies (Nelson

et al., 1992):

The bricoleur produces a bricolage, that is a pieced-together close-knit set of practices that provide

solutions to a problem in a concrete situation. The solution (bricolage) which is the results of the

bricoleur’s method is an emergent construction (Weinstein and Weinstein, 1991: 161; cited by

Denzin and Lincoln, 1998: 3).

In this case, the bricolage is the solution to a whole collection of evaluation needs that exist for one

large project. Figure 3 illustrates how a bricolage may be developed. The idea is that firstly the

existing evaluation and current practice should be audited. Following this, an investigation into the

33

needs of the stakeholders of the evaluation is conducted. This stage includes identifying and

negotiating with the evaluation audiences, determining resource availability, and if necessary,

drawing a program logic diagram to understand how the project is believed to meet needs. These

facets together allow an identification of evaluation needs and gaps. From this, the intention is to

select appropriate new approaches, and to trial and modify these to meet the evaluation needs of the

project. This process is cyclical; the whole cycle may have to be run several times in the course of a

project as evaluation needs may change.

Figure 3 The process of modifying an evaluation bricolage for a project

The idea of an evaluation bricolage is to build on existing evaluation practice and project

experience. Therefore, the aim of this thesis is to develop one new piece of the bricolage – that is,

an approach that can, in combination with other approaches, address the current challenges

associated with evaluating the new genre of agricultural extension projects in Australia. An

essential part of developing a new approach to evaluation is that it should be piloted and modified

to suit the particular program context, and to determine whether it does indeed meet needs, and

augment the existing evaluation bricolage.

Identify information and current evaluation practice (understand the existing bricolage)

Identify evaluation needs of the project

Choose or develop a new evaluation approach to meet needs/gaps

Pilot the new approach

Evaluate the new approach (meta-evaluation)

Review and revise the bricolage as evaluation needs are clarified and change

Determine the evaluation audience

Establish the resources available

Negotiate with evaluation audience

Clarify the program logic if necessary

34

2.6 A Normative Framework for Evaluation

Alkin et al. (1979: 25-26), in their review of evaluation literature, point out that different definitions

of evaluation lead to different notions of ‘good evaluation’. Clearly, depending on the normative

approach being adopted, different criteria can be applied in determining what constitutes good

evaluation. Still, it is possible to combine some of the features from all six approaches reviewed to

develop a useful guide for determining what constitutes good evaluation.

The following paragraphs outline a series of criteria that I suggest should be present in evaluation. I

describe eight premises, which I then re-phrase as meta-evaluation questions. In developing these

premises I have drawn strongly on the examination of the different normative approaches for

evaluation.

Premises for good evaluation

Firstly, good evaluation is relevant to the context and purpose in hand. Thus, evaluation models for

agricultural extension should consider the environment in which the program is being conducted

and the evaluation needs of the stakeholders. This view is strongly advocated by Patton in his book

Utilization Focused Evaluation (1997). For this reason, Chapter 3 presents an examination of the

context in which agricultural programs are planned, delivered and evaluated and Chapter 4

examines current practice in agricultural extension in Australia.

Secondly, as advocated by Rogers, (1996) good evaluation contributes to improved projects; that is,

projects which meet needs. Rogers credits this premise to Madaus, Stufflebeam and Scriven

(1983):

Ultimately the value of program evaluation must be judged in terms of its actual and potential

contributions to improving learning, teaching and administration, health care and health, and in general

the quality of life in our society (1983: 18).

Improved projects can come about in one of three ways; improving existing projects, developing

alternative projects, or developing new projects. The implication of this is that good evaluation is

able to show evidence that the evaluation did contribute to project improvements.

The third premise, borrowed from theory-driven evaluation, is that good evaluation considers the

theories underpinning the project (Chen, 1990). Considering the arguments of Pawson and Tilley

35

(1997) and Chen (1990), evaluation should be guided by program theory, and should not be simply

a black box venture. Even Scriven (1994), who levelled early criticisms of theory-driven evaluation

has recently conceded that while program theory may not always be essential, it is usually helpful.

The fourth premise is that good evaluation seeks to understand the differentiated nature of program

effects including any unanticipated outcomes. Pawson and Tilley (1997) stress that it is important to

look beyond whether a program works or not, and to consider the extent to which outcomes are

realised in different contexts. Scriven’s work on Goal-free Evaluation serves to remind us to look

beyond the program objectives and to also investigate unexpected outcomes. Whether an

appreciation of differentiated program effects is achieved through collecting the different

constructions of reality, (as in pluralist-intuitivist approaches) or by generating and testing theory

about how a program works in different contexts (as in a theory-driven approach), some notion of

the stratification of evaluation findings should be included. A purely positivist stance does not serve

evaluation well and may subsume genuine differences in findings in an attempt to present an answer

to the question, does the program work or not?

Fifthly, even if the focus on the program is not explicitly upon social justice, any process of

evaluation should be socially and politically just, and conducted in an ethical manner. The views of

all people affected by the project should be sought, rather than giving privilege to one potentially

dominant group. In good evaluation attention is paid to who is given voice in the determination of

the indicators for the evaluation. This is stressed strongly in some of the pluralist-intuitionist

approaches, particularly Guba and Lincoln’s (1998) Fourth Generation Evaluation.

Sixthly, the time and resources put into an evaluation should not outweigh the benefits. This is in

recognition of the need for what has been termed ‘real time’ evaluation. If evaluation findings are

intended to influence practice and policy, then they must produced in time to be useful to decision-

makers. Re-structuring and change seem to be endemic in organisations nowadays, so it is

imperative that evaluative information is timely, otherwise it will be dated by the time it is received.

Again the resources expended should be weighed against the benefits expected from an evaluation

study, for example for a quick snap shot evaluation, a detailed investigation of underlying causal

relationships (as in theory-guided approaches) might be uncalled for.

The seventh point is that, for any model of evaluation, the design, data collection and analysis of the

evaluation must be valid. This presents problems in defining criteria for validity that are equally

36

applicable across different paradigms. To some extent, each underpinning paradigm will come with

some notion of good design and data collection. For example, with the Fourth Generation

Evaluation model, good evaluation involves including as many relevant constructions of the

situation as possible. For the realists, good evaluation might concern developing theory that can be

most closely correlated with reality. For advocates of Utilization Focused Evaluation it might be

that program staff use the evaluation findings. So the premise here is that good evaluation employs

a design and data collection that is valid under the paradigm associated with the evaluation

approach.

The eighth point, strongly stressed by Scriven, is that evaluation should attempt to judge the merit

and worth of the project. Who it is that should be making this judgement will depend on the purpose

of the evaluation. It may not be the evaluator that makes this judgement, such as in the case of

participatory approaches to evaluation.

Meta-evaluation questions summarised

The eight premises for good evaluation can be reworded as meta-evaluation questions (see below)

These meta-evaluation questions will be used to guide the review of current evaluation practice

(Chapter 4), and for the meta-evaluation of the novel approach that is tested as part of the case study

(Chapter 8). The meta-evaluation questions are:

1. Is the evaluation relevant to the context and purpose in hand, and does it fulfil the evaluation

needs of the stakeholders?

2. To what extent can the evaluation contribute to the project being better able to meet needs?

3. Is the evaluation guided by program theory?

4. Does the evaluation make an attempt to account for different program outcomes?

5. Are the evaluation processes socially and politically just?

6. Is there an appropriate ratio of costs to benefits for the project from the process (and outcomes)

of the evaluation?

7. To what extent are the design, data collection and analysis of the evaluation valid?

8. Does the evaluation attempt to judge the merit and worth of the project?

37

Definition of evaluation

From the premises outlined above, and the discussion of normative models of evaluation, an

overarching definition of evaluation was developed adopted in this thesis. It is:

Program evaluation concerns the systematic collection of information, in order to improve

decision making and enhance organisational learning, with the ultimate aim of bringing

about programs that better meet needs and lead to amelioration of targeted social, economic

and environmental problems.

2.7 Conclusions

In this chapter an attempt was made to make sense out of the diverse and conflict-ridden field of

program evaluation. Over the last 30 years there has been an explosion of new models advancing

theory as to how best to conduct evaluation. For this thesis, a pragmatic stance is adopted, in which

I suggest that the choice of evaluation model should be grounded in knowledge of the program

context, and of the key evaluation task. In line with this, the following two chapters will examine

the needs within the context of agricultural extension in Victoria (Chapter 3) and the existing

evaluation practice in Australia (Chapter 4).

I suggest that new approaches should be selected (or developed) with an understanding of the

different approaches of program evaluation available, and with some notion as to what constitutes

good evaluation. New approaches should be developed through field-testing and meta-evaluation,

against the contextual needs of the organisation. There should be an understanding of how the new

approach attempts to contribute to bringing about improved programs that meet stakeholder needs,

and that ultimately contribute to ameliorating social, economic and environmental problems.

For this reason, a series of meta-evaluation questions were developed to guide the formation of a

new approach. Chapter 7 describes how a novel evaluation approach was modified, and

implemented across a Victorian extension project. In Chapter 9 this approach is evaluated using the

meta-evaluation questions developed. In this way the framework contributes to achieving the aim of

this thesis, which is to develop a model of evaluation for program evaluation of Australian

extension.

38

CHAPTER 3

EVALUATION NEEDS IN AGRICULTURAL EXTENSION

In this chapter I present an overview of evaluation needs in Australian agricultural extension

projects. Emerging demands for evaluation are linked to changes in both the philosophy of

extension and changes in the structure of extension institutions. In the last 10 years, Australia

has seen major changes in the way the Government manages public sector programs. There

have also been changes in the philosophy that underpins agricultural extension and

associated changes in extension practice. These changes have influenced the way publicly-

funded extension projects are planned, managed and valued, and present unresolved

dilemmas with regard to evaluating the new genre of projects. An analysis of the changing

nature of extension revealed the need for evaluation that can deal with participatory, group-

based extension, and that can identify and value diverse impacts. An analysis of the changing

organisational environment of extension revealed there is an increased demand for internal

evaluation that can deal with multiple audiences, and promote organisational learning and

communication, as well as meeting needs for accountability.

3.1 Introduction

Globalisation and new technology have led to changes that seem to affect every facet of life.

Agricultural extension is not immune to these changes and currently Australian extension

institutions are undergoing considerable structural and philosophical changes. Accordingly, changes

have occurred in the sorts of evaluation that are being demanded for these projects. To develop

effective approaches to project evaluation in these changing times, an appreciation must be gained

of the environment in which agricultural extension is delivered and evaluated. The aim of this

chapter is to describe the current context of agricultural extension in Australia and to identify the

implications for the practice of evaluation within this field. To achieve this, two main areas of

change are examined. The first addresses changes to the philosophy and theory underpinning

agricultural extension itself; the second incorporates changes affecting the organisations in which

39

extension is delivered.

I begin the chapter with a brief look at changes that are occurring at a global level and how these

changes are reflected in the practice of agricultural extension. I provide a description of the

development of extension models from the ‘science-push’ paradigm, to new approaches influenced

by the emergent ‘farmer-first’ paradigm. I emphasise the move towards more participatory, group-

based and process-oriented forms of extension. Following this, I examine the changes occurring in

the organisations that deliver extension. As each state of Australia has a different story to tell, I take

an in-depth look at the case of Victoria. In doing so, I focus on the impact of recent reforms in the

Agriculture Division in the Victorian Department of Natural Resources and Environment, and

specifically on the provision of agricultural extension. I make an attempt to unravel the various

reforms and trends occurring and describe the consequent challenges posed both for the practice of

agricultural extension and of project evaluation in agricultural extension.

3.2 A Changing World

Increasing globalisation, improved communications, and new technologies have resulted in growing

social, political and economic interdependence at a global level. Even small shifts within society are

felt throughout the world. Stock market jitters in Tokyo are reflected in London and New York in a

number of hours. Familiar patterns of family, community, business, politics and even nation are

breaking down. At the same time there is something else new in the air, a new emphasis on unity

and integration, talk of new politics, even a new society. The old ‘modernist’ or positivist vision of

one truth, one expression of reality, one best way of doing things seems problematic in this

globalised world. Instead, there is a need for a more pluralistic vision that can accommodate the

multiplicity and diversity of experience.

There are several features of this new global rhetoric that cross many disciplinary boundaries and

countries. From agricultural extension (Chambers et al., 1991), to quantum physics (Zohar and

Marshall, 1994), to the ‘third way politics’ (Giddens, 1998) to action research (Reason, 1988) and

to program evaluation (Guba and Lincoln, 1989). Four main elements recur in this globalised

discussion, i.e, things must be done in ways that are:

40

• Holistic – due to increasing globalisation and improved communications we must be aware of

the interconnectedness nature of interventions. There is a tendency to encourage collaboration,

partnerships, and dialogue between organisations that were previously perceived as separate.

• Pluralistic – due to increased globalisation and collaboration we must learn to live with many

points of view, many different ways of experiencing reality.

• Responsive – new systems must be flexible and less hierarchical. Ambiguity, rapid change and

greater complexity increasingly dominate events and patterns of relationship in both domestic

and public life. New family structures, new technological systems, new and shifting markets all

demand flexible responses.

• Bottom up or emergent – increasingly citizens, have grown impatient with political structures

that are implemented in a top-down manner. More citizens are being invited to join in the

decision-making processes.

This global rhetoric also appears in texts associated with the practice and philosophy of agricultural

extension, where the traditional top-down, ‘science-push’ approach to extension is largely being

replaced by extension that is described as holistic, responsive, pluralistic and bottom-up. At the

same time this global rhetoric also appears in literature associated with public sector management.

Indeed, the public sector organisations that have traditionally delivered extension have been subject

to many changes associated with a new type of global public sector management. For example, in

extension organisations in Australia there is a tendency to move toward organisations with fewer

hierarchical levels, emphasising new partnerships with the private sector, and with more farmer

representation at many levels of the organisation.

3.3 Changing Philosophy of Agricultural Extension

In this thesis, I refer largely to publicly-funded extension, while acknowledging that there is

increasing involvement of the private sector in the provision of advice and other information to

farmers. Ultimately, governments fund extension to enact policy. Because of this, extension

changes with the uses to which it is put and the historical context in which it is deployed (Röling,

1988). Changes associated with the paradigms that underpin extension are influencing how

extension is conceptualised, how it is conducted, and subsequently how it should be evaluated.

41

In the following sections, the concept of extension will be examined followed by a discussion of the

movement from a worldwide adoption of the ‘science-push’ paradigm in extension, to the

emergence the ‘farmer-first’ paradigm, and finally to the current situation in Australia where there

is no idealised model of extension in practice.

Conceptualising extension

Before exploring the changing nature of the paradigms that underpin extension, an overview of the

range of things that can be signified by the term ‘extension’ will be provided. At the heart of

extension is the contradictory notion of inducing behaviour change by voluntary means, rather than

by regulation. Behaviour change is a complex thing to enact on oneself, and even more so to enact

on others. Inducing voluntary change is generally only successful when the target person’s goals

are also satisfied in some manner. Differences in the way that extension is conceptualised tend to

reflect the tension between inducing behaviour change and satisfying client goals. As a result of

this tension, extension is a messy and ill-defined concept.

Röling (1998: 36-37) points out how in different countries extension has been conceptualised and

labelled quite distinctly. For example in French it is refereed to as vulgarisation: a simplification of

information so that ordinary people can understand; in Dutch it is referred to as voorlichting:

keeping a light in front of someone to allow him or her to find their way; in Spanish it is referred to

as capacitación: empowering someone to achieve things for themselves. Van Dissel (1986)

contends that as a result of the confusion, extension has never established an international

classification of extension publications. This helps to explain why there is no uniform definition of

extension. Zuurbier (1983; cited by Röling, 1988: 39) examined the various definitions of extension

across languages and literature and determined five common elements:

• extension is an intervention

• extension uses communication as its instrument to induce change

• extension can be effective only through voluntary change

• extension focuses on a number of different target processes and outcomes, which distinguish it

from other communication interventions

• extension is deployed by an institution.

42

These common elements provide an overarching description for extension. However, it is also

useful to understand the differences between alternative extension traditions. Röling (1988: 37)

describes four traditions, each of which are present to a lesser or greater extent in different cultures

of extension:

• Informative extension – where the emphasis is on providing information to help the individual

to make optimal decisions with respect to achieving her/his own goals.

• Emancipatory extension – where extension is seen as an instrument of emancipation of the

poor, as advocated by Friere (1972). This sort of extension is most usually carried out by

voluntary agencies who seek to assist the rural poor, often in low-income countries.

• Human resource extension – where the emphasis is on the development of human beings in

terms of enhancing her/his capability to make decisions, to learn, to manage, to communicate

with others.

• Extension as a persuasion – where the emphasis is on extension as a policy instrument for

achieving societal objectives or collective utilities.

In Australia, much of the publicly-funded extension appears to fit in with Röling’s tradition of

extension for ‘persuasion’, with export goals, national food security and environmental protection

being the driving issues. However, for farmers to be ‘persuaded’ into voluntary change, farmers’

goals must also be satisfied. As a result of this, many extension projects appear to come in various

combinations of Röling’s four traditions.

A frequently cited definition of extension is that of Van den Ban and Hawkins who define extension

as ‘the conscious use of communication of information to help people form sound opinions and

make good decisions’ (1988: 9). However, this definition provides no clues as to who decides what

constitutes ‘sound opinions and good decisions’. In publicly-funded extension, the assumption is

that ‘good decisions’ constitute those decisions that lead to behaviour change congruent with the

current government policy. The idea behind government funding for extension is to bring about

mainly public rather than private benefit. While public and private benefits often overlap, it is

possible that a publicly-funded extension activity may aim for voluntary change in the behaviour of

a farmer, which may not necessarily benefit that individual farmer at all. This illustrates the

ambiguous nature of the term ‘extension’.

Historically, the rhetoric of agricultural extension worldwide has shifted from an emphasis on

43

production, at the beginning of the century, to productivity (or efficiency) based agriculture, to the

more recent philosophy of sustainability (Guerin and Guerin, 1994). Pretty (1995) suggests that we

are currently entering an era of social capital, where farmers are now considered to be the potential

solution rather than the problem, i.e., the role of individual capacity is paramount. Through each of

these eras, adoption of relevant technologies and innovations by the farming community has been

critical to improve and maintain agricultural production and to meet changing demands (Campbell,

1980). However, the way in which adoption is brought about, and the way in which innovations are

developed has been widely contested. Two major contending paradigms are:

a) the ‘science-push’ paradigm that relates to Röling’s description of extension as persuasion

b) the ‘farmer-first’ paradigm that relates to either Röling’s description of extension as human

resource extension or informative extension.

The ‘science-push’ paradigm of transfer of technology

Under the reductionist paradigm of ‘transfer of technology’ (TOT), agricultural extension is seen as

a vehicle to transmit research findings to farmers, who then adopt the superior technology which in

theory leads to more efficient and effective production. This view of extension can be described as

‘science-push’, as research results are extended from scientists to the farmer in a linear manner.

Under this model, highly skilled and trained scientists set the research agenda and extension is

usually subservient to research. Engel and Van den Bor (1995: 2) state that this paradigm ‘led us to

believe that extension is an institution which merely channels knowledge without adding value to

it’.

A central concept behind the TOT model was Roger’s (1962) ‘diffusion of innovations’ theory,

whereby extension officers would transfer the latest research findings and technologies to

innovative farmers. The theory was that this would trickle down (diffuse) across the whole farming

community. An emphasis was placed on developing agricultural technologies on research stations,

and bringing about mass adoption of these technologies as quickly as possible to maximise

productivity gains (Coutts, 1997). Although social aspects are considered under the TOT model,

extension was mainly conceived in the context of increasing the rates of adoption. Thus, projects

governed by the TOT paradigm are traditionally evaluated against the extent to which the new

technology is adopted and sometimes the consequent change in production. Despite its prevalence,

its success is said to have been limited to homogenous groups of farmers who already have a high

44

level of access to information and resources or to homogenous groups that have been specifically

targeted (Röling and Jiggins, 1987).

The ‘science-push’ paradigm remains the basis of much research and advisory structure in Australia

(Guerin and Guerin, 1994). On a global level, the 1970s witnessed the beginning of disillusionment

with the TOT model. Much has been written on the shortfalls of the TOT and the ‘science-push’

approach (Chambers, 1983; Dillon, 1965; Vanclay and Lawrence, 1995). The central arguments are

that:

• technologies developed on controlled research stations did not diffuse across the farming

population as expected

• the benefits of the research tended to be distributed inequitably across the farming community

• the products of agricultural science were uncritically accepted by researchers as improvements

• the focus on disciplinary research was unable to provide solutions to problems at the whole

farm level, or for complex farming systems

• the emphasis on production overlooked issues of environmental and social sustainability

• the emphasis on the supremacy of scientific knowledge overlooked local or indigenous

knowledge.

Despite the fact that ‘science-push’ model is still alive in Australia (Guerin and Guerin, 1994), in

the last 30 years the farmer-first paradigm has emerged, and has led to a new family of approaches.

These new approaches include: Farming Systems Research (FSR), Farmer Participatory Research

(FPR), Participatory Technology Development (PTD), farmer-first research, Participatory Action

Research (PAR), co-learning and experiential learning. All of these approaches emphasise farmer

involvement in the identification and testing of suitable innovations or ideas to improve their

situations.

The emergence of the farmer-first paradigm

The 1970s saw the emergence of a new interpretation of the concept of non-adoption. Instead of

non-adoption being considered an issue of ‘farmer failure’, it became re-conceptualised as an issue

of the inappropriateness of recommended technologies or the priorities and processes which

generated them (Chambers et al., 1991). There is now considerable agricultural research and

development activity that was instigated on the basis of claims that farmer participation is critical to

45

the generation of technologies that are relevant to farmers (Bebbington et al., 1994; Foale, 1997).

Chambers contends that:

With farmer-first, the main objective is not to transfer known technology, but to empower farmers to

learn, adapt and do better; analysis is not by outsiders – scientist, extensionists, or NGO workers – on

their own but by farmers and by farmers assisted by outsiders; the primary location for R&D is not the

experiment station, laboratory or greenhouse…but farmers’ fields and conditions; what is transferred

by outsiders to farmers is not precepts but principles, not messages but methods, not a package of

practice to be adopted but a basket of choices from which to select (1999: 182).

In Australia similar concepts emerged during the 1960s and 70s (Williams, 1968; Bardsley, 1981).

Researchers were seen to have more of a support role, and extension had more of a facilitation role

than in the past. The shift towards the farmer-first paradigm is associated with several related

approaches to extension, three of which are presented in the following sections.

Systems based approaches to extension

In the 1970s Farming Systems Research (FSR) was developed to address some of the constraints of

the TOT model. This systems approach combines elements of research with extension. FSR

considers the whole farm rather than studying individual components, is conducted on the farm as

opposed to the research station, and ideally involves farmers in defining the research agenda. The

adoption of farming systems research at most international agricultural research institutes has had a

large impact on national projects in many countries (Anderson and Dillon, 1985). In Australia,

systems based models of research and development have been recommended or applied in some

regions (see McCown, 1991; Clark et al., 1996; Foale, 1997) although the approaches vary widely.

It has been suggested that farming systems research has made a huge contribution to the ‘farmer-

first’ paradigm by revealing the complexity of farming systems and of the decisions which face

farmers, the limitations of disciplinary research, and the recognition of farmers as innovators and

experimenters themselves (Chambers et al., 1991; Rhoades and Boothe, 1982; Petheram and Clark,

1998).

Experiential learning, co-learning approaches and action research

The Hawkesbury model of agricultural systems development (Bawden, 1991; Macadam, 1996) is

based on experiential learning (Kolb, 1984), which encapsulates the concept of learning through

experience. One way it differs from FSR is that it emphasises the human activity system, rather

than the physical/biological system (Petheram, R.J., pers. comm., 2000). Kolb (1984) defines

46

experiential learning as the process whereby knowledge is created through the transformation of

experience. The cycle of experiential learning is based around a cycle of concrete experience,

reflective observation, abstract conceptualisation and active experimentation.

Participatory Action Research (PAR) or Action Research is similar in many ways to experiential

learning models. PAR, as its name implies, pursues both action and research outcomes. It is most

commonly done with high levels of participation of farmers and agency. Reason (1988) suggests

that PAR is probably the most common collaborative research approach. Its purpose is to bring

about planned change in such a way that there is also a contribution to understanding of the system

which is being changed, and the methods which are used. Hart and Bond (1995) selected seven

criteria to distinguish action research from other research methodologies. The criteria are that action

research:

• is educative

• deals with individuals as members of social groups

• is problem focused, context-specific and future-oriented

• involves a change intervention

• aims at improvement and involvement

• involves a cyclical process in which research, action and evaluation are interlinked

• is founded on a research relationship in which those involved are participants in the change

process.

These criteria could equally be applied to experiential learning, another co-learning model that has

been applied in Australian contexts. Indeed Roberts (1997) suggests that action research may be

viewed as connected cycles of experiential learning and that action research and experiential

learning may be interpreted as a single phenomenon described from different perspectives. All these

experiential approaches have reflection and evaluation phases built into their cycles. Experiential

approaches produce context-specific outcomes, which will differ greatly from one project to the

next.

47

Group-based approaches to extension

In Australia, particularly with the advent of Landcare, there has been an increasing use of group-

based approaches to extension. Before the 1980s most extension was either done on a one-to-one

basis or at field days, or through mass media. Coutts states that:

From my perspective the appointment of Landcare and Catchment facilitators was the greatest single

challenge to extension officers operating in the traditional mode of one-on-one advice. These

facilitators came in without the strong technical background of extension officers – and without the

baggage of associated with the extension culture. They demonstrated a new paradigm in working with

groups of producers (Coutts 1997: 25).

Community-based groups could well form the basis for a trend towards research, extension, and

advisory systems that are bottom-up or (farmer-first), rather than linear (Campbell and Junor, 1992).

The idea of giving legitimacy to local knowledge and of working with groups, demands new skills

in those involved in extension, and in those evaluating the effort.

Multiple approaches to extension

In the year 2000, there is no one clearly idealised model of extension being practised in Australia.

Coutts (1997) describes extension as slipping into a range of purposeful and complementary

models, shaped by political and funding realities, rural and societal needs, and a greater

understanding of communication, facilitation and adult learning. Vanclay (1994) suggests that

Australian public extension is facing a crisis of effectiveness and theory, with extension practices

not working, farmers failing to adopt many recommended practices, and agencies rejecting the

traditional extension model without a cohesive, coherent or widely accepted alternative. Whether

viewed in a positive or negative light, the above literature indicates that there is no single theory

currently underpinning current practice.

While acknowledging many exceptions, the late 1980s to 1990s saw the emergence of two different

approaches with regards to planning and managing agricultural extension projects in Australia. On

the one hand, some projects became increasingly targeted towards productivity, aiming to achieve

pre-stipulated export targets. Projects were given names reflecting the export targets that had been

stipulated at the state level, i.e., ‘Target 10’, ‘Farm Management 500’, and ‘Target 250’. On the

other hand, a movement emerged calling for the need to embrace the concept of ‘learner focused’

48

extension. This movement embraces many of the concepts that underpin the farmer-first paradigm

and the co-learning and experiential approaches. These projects such as ‘Farm$mart’, ‘Topcrop’

and ‘Bestwool’, aim to help the farmers to help themselves. Projects are often organised around

farmer groups that meet regularly, where participants identify critical issues for learning and action.

While these approaches to planning sound contradictory, they often sit side by side within the same

project. For example, the Victorian Target 10 Dairy Extension Project is strongly outcome-focused

and has an explicit target to increase the production of milk. However this project is currently

attempting to move towards a more learner-focused approach, through discussion groups and wide

representation of farmers of steering committees and in project redevelopment (Boomsma, J. pers.

comm., 2000).

The implications for evaluating new approaches to extension

Changes in the philosophy of extension have seen the emergence of new approaches to extension.

The following characteristics of these new projects have practical implications for evaluation, and

will be discussed in the next section. These are:

• increasing participation of farmers in the evaluation of extension activities

• increasing process-orientation

• increasing diversity in outcomes

• increasing complexity in the scope of projects

• increasing number of organisations working in extension.

Increasing participation of farmers in the evaluation of extension activities

Worldwide, development and extension projects are becoming more participatory in focus. The

past two decades have seen an increased recognition of the importance of participation by

beneficiaries (and a wide range of other stakeholders) in decision making. There has also been

increasingly wide and deep discussion about the theory of participation appropriate levels of

participation (Pretty et al., 1995). For extension projects operating at either end of the participation

continuum, increasing calls for accountability, efficiency and improved management performance

have led to a more intense focus on monitoring and evaluation. But in participatory projects three

additional questions emerge:

49

• whose reality counts?

• who decides what is measured in the evaluation?

• who should be making the evaluative judgements?

One of the major challenges facing the field of international development in the 1990s has been

how to measure the impact of participatory projects in a manner that is congruent with the

philosophy of these projects (Oakley et al., 1998). The over-riding concern is for the process of

evaluation to reinforce, rather than inhibit, participation and empowerment of beneficiaries.

Clearly, external evaluation, based on outsider values as to what constitutes success, would not be

appropriate. In short, participatory projects require participatory evaluation; approaches that allow

the stakeholders and beneficiaries to have a say in which changes are important, and which changes

should be measured. Patton, writes that participatory evaluation:

…is most appropriate where the goals of the project include helping participants become more self-

sufficient and personally effective. In such instances...evaluation is also intervention orientated in that

the evaluation is designed and implemented to support and enhance the program’s desired outcomes

(1997: 101).

Evaluation that can meet these requirements is referred to as ‘participative’ or ‘participatory

evaluation’, or ‘participatory monitoring and evaluation’ (PM&E), or in the US, ‘empowerment

evaluation’. In this thesis this genre of evaluation is referred to as participatory evaluation.

Participatory evaluation promotes the involvement of a wide range of stakeholders, employing

methods that encourage an equal expression of views and sharing of lessons.

Increasing process-orientation

The idea of process-orientation in extension projects was first seen 20 years ago in the field of

overseas development projects. Farrington and Nelson (1997) suggest that in agricultural

development projects in lower income countries, there is an increasing tendency towards

participatory projects with a ‘process-orientation’. The term ‘process-orientation’ in this context is

used to describe projects that do not have rigidly defined goals at the onset but have a defined

purpose and a set of potential goals. The concept of process-orientated projects emerged while

attempting to encourage ‘bottom-up’ projects. This was problematic when predicted outcomes and

activities were pre-stipulated in a ‘top-down’ manner. Process-projects are a natural evolution of

the concept of beneficiary participation in project planning and decision-making and defining

intended outcomes. If beneficiaries are to identify their development needs as part of the project

50

process, then it makes sense to have a project design that does not have predetermined outcomes.

The Landcare movement in Australia, (although it developed independently from development

studies literature) exhibits properties of process-oriented projects.

Farrington and Nelson (1997) suggest that projects with a process-orientation ‘are difficult to

evaluate by traditional means as they are carried out in variable, unpredictable situations; they

produce outputs that are hard to measure objectively and often have permeable boundaries and less-

than-direct relationships between inputs and outputs’. Evaluation of this type of project is best

achieved though inductive, participatory methods, where dialogue and exploration are encouraged.

Objectives-based evaluation is unlikely to capture the variable impacts of projects; instead

exploratory, participatory, needs-based evaluation is more appropriate.

Increasing diversity in outcomes

Another consequence of the new paradigm approaches to extension is recongition of diversity.

Farmer-first approaches tend to be contextual in focus and are not designed to produce generalisable

results. Farmers are encouraged to seek solutions to their location-specific problems. As a result

there is often a great variety in terms of the experiments conducted by the groups. These projects

tend to produce diverse, context-specific outcomes, which are difficult to aggregate in order to

provide information to evaluate the project as a whole.

Conventionally, planning and evaluation of simple extension projects have been managed through

the use of simple project models in which planned inputs lead logically to predictable outputs or

outcomes. When project models become more complex and flexible it becomes imperative to

expend effort in investigating the unintended or unanticipated outcomes. When evaluating complex,

diverse projects, it becomes critical to positively engage with unpredicted or unanticipated impacts

and diversity, rather than just an extra task to tack to the end of an evaluation study.

Increasing complexity in the scope of projects

The information/knowledge system (IKS) of agricultural extension in Australia is becoming

increasingly complex. The shift in focus of extension from production orientation to a holistic

view, including: sustainability, improved marketing, increasing attention to the management of the

farm business and maintaining profitability under declining terms of trade, adds to this complexity.

51

Increasing number of organisations working in extension

There are more actors operating in the capacity of knowledge bearers to farmers than in the past.

The implication of this increasing complexity of the IKS on evaluation is that it is becoming more

difficult to distinguish the impact of one project or actor from that of another (Coutts, 1997).

Evaluation studies need to take into consideration the implication of the integrated impact of the

workings of the IKS on the rural sector.

Summary of implications of evaluating new paradigm approaches to extension

The evaluation of participatory and process-orientated projects must take into account that the

outputs of such projects tend to be diverse, unpredicted and context-specific. Conventional,

objectives-based approaches are inadequate to capture the range of unpredicted and unanticipated

outcomes that these projects produce. In addition, external evaluation against pre-defined indicators

of success would be incongruent with the philosophy of these new paradigm approaches.

Nonetheless, these projects experience the same demands for accountability and demonstration of

impact as all publicly-funded extension projects.

Allen (1998: 629) contends that because participatory projects are designed to be responsive to

changing community needs, one of the most pressing challenges is to develop ‘participatory and

systems-based evaluative processes to allow for ongoing learning, correction and adjustment by all

parties concerned’. Alternative methods are needed for evaluating participatory, group-oriented

approaches to extension; as participatory projects require participatory evaluation. If evaluation

approaches do not exist within the current repertoire of extension evaluation tools, they need to be

developed or adapted from other disciplines. These methods need to be participatory, creative,

formative and empowering. Chapter 4 reviews the current practice of evaluation in extension to

determine whether any of the approaches already being practised in this field, meet such criteria.

52

3.4 Organisational Change and Extension Evaluation

Historical changes in the practice of evaluating extension in Australia

As discussed in Section 2.2, agricultural extension was one of the first disciplines to conduct

program evaluation. As early as 1947, some major projects in Australian agricultural extension

were evaluated as part of their planning and funding process (Sharp and Lindsay, 1992). Over time,

the purpose and methods of extension evaluation have been strongly influenced by the

organisational environment in which farmers and extension agencies have had to operate. A key

element in this environment has been government policy. Changes in federal Government policy

and corresponding changes in evaluation practice can be loosely characterised by decades, although

it must be acknowledged that these changes occurred somewhat differently in different states of

Australia, each with unique state elected governments.

In the 1960s and early 1970s agricultural extension was seen as a mechanism to solve farmers’

problems, and with this view evaluation was used as a tool to provide feedback, to improve the

services to farmers. The approach characterised the optimistic nature of this period. Evaluation was

much discussed at extension conferences, and national workshops were conducted to focus attention

on taking a more professional approach to evaluating the efficiency and effectiveness of extension

(Hynes, 1963). Issues such as how to measure behavioural change and the need for open-ended

questions were addressed (Riches, 1973). Bardsley (pers. comm., 1997) recalls that in the optimism

of the period, extension workers felt valued by their farmer clients.

Bardsley (pers. comm., 1997) suggested that during the 1970s under the Whitlam Government,

extension became progressively more of a political tool used to bring about policy changes desired

by government, than previously. There was also a growing need to justify the spending of public

funds. With this change in perspective towards agricultural extension came a need for evaluation

methods to be more centred around accountability. Much of the evaluation carried out at this stage

was based upon assessing the achievement of institutional goals or the achievement of objectives of

individual extension officers (Sri Pathmanathan, 1978). Little attention was paid to whether the

clients’ needs were being met by the project (Bardsley, 1981). To an extent, agricultural extension

became more of an arm of government policy and the resulting projects were characterised by a

53

focus towards national interest, which was sometimes at odds with individual farmer interest.

In the 1980s governments began to question whether agricultural extension should be publicly-

funded and thus emerged the principle of user pays. At the Australian Agricultural Extension

Conference in Brisbane in 1987, the question of funding government extension services received

considerable attention. The majority of the discussion concerned the application of ‘user pays’ as

this was seen as the most effective way to overcome the decline in extension funding. With this

shift towards a decrease in public spending in agricultural extension, came a greater demand for

evaluation of both publicly and privately-funded projects. Private enterprises wanted to make sure

that projects, which were funded with private money, were effective and accountable. Thus began

the trend for evaluation to be used as a tool for justification and accountability of spending.

In the 1990s, for some states of Australia, the funding for extension appeared to stabilise, and the

prospect of a totally user-pays system seemed to fade (Coutts, 1997). However, a user-pays system

was actually implemented for the provision of agricultural extension in Tasmania for a period,

although it is now reported that some aspects of extension are being recognised as public benefit

and are attracting public funding (Coutts, 1997).

The 1990s was increasingly influenced by changes that were occurring at a global level and referred

to as ‘New Public Management’ (Hood, 1991). These reforms included the introduction of the

purchaser-provider model for publicly-funded agricultural extension in many states of Australia

(Marsh and Pannell, 1997). This is one of a plethora of recent political reforms that have been

introduced in a more general attempt to mimic free market conditions. These reforms, which are a

worldwide phenomenon, pose considerable challenges for the evaluation of extension projects and

projects. The reforms affect both the product of extension and the expectations of how this product

should be evaluated and these are discussed in more detail in the next section.

Reforms and new public management

Much of the literature concerning New Public Management (NPM) is from other sectors and other

countries, and does not specifically refer to agricultural extension. However, as I will describe, the

literature of NPM adequately addresses the changes occurring for some of organisations that

provide extension in Australia – changes that are part of a global phenomenon. The last 10 years

have seen major changes to the way public sectors are managed in much of the Western world.

54

These changes have variously been referred to as ‘New Public Management’ (Hood 1991) ‘New

Managerialism’ (Pollitt, 1997) and ‘Contractualism’ (Alford and O’Neill, 1994). These reforms

have involved creating a more ‘business-like management’ of the public sector (Lane, 1997).

Martin and Sanderson (1999) suggest that NPM has typically consisted of:

…a cocktail of cost reductions, devolution of management responsibilities, ‘neo-taylorist’ performance

management system, new market and quasi-market mechanisms, greater ‘customer focus’ and new

service standards. Taken together these reforms represent a decisive shift away from management

based on tight ex-ante control of inputs, towards a framework which devolves responsibility for service

delivery to front-line staff within a system of ‘continuous monitored management by objectives with

accountability for results (ibid: 246).

O’Faircheallaigh et al. (1999: 4) suggest that under this New Public Management, governments of

OECD countries are calling for less ‘red tape’, more emphasis on results, greater efficiency,

reduced staff and user-charging for services in public agencies. Governments are increasingly

asking whether they should remain involved in or withdraw from the activities that they presently

undertake. Rivlin (1996, cited by O’Faircheallaigh et al., 1999: 1) suggest that these new public

management models attempt to deal with problems arising from:

• globalisation – global pressures to cooperate and compete in new ways

• dissatisfaction – ever rising expectations of citizens

• budget stringency – the need to curtail expenditure and reduce deficits.

O’Faircheallaigh et al. (1999) provide a useful framework for unravelling and describing the

plethora of changes that are occurring under New Public Management worldwide. They consider

the reforms in terms of the disciplines that influenced them, namely: economics, organisational

theory and management, political science, and marketing. In the following sections

O’Faircheallaigh et al.’s framework is summarised, then applied to the particular case of

agricultural extension in Victoria, in attempt to understand the emerging implications for program

evaluation in this sector.

Influence of economics on public sector management

O’Faircheallaigh et. al. (1999: 36) explain how economics has helped shape many of the tenets of

NPM. Economics, as practised in many western countries, has long been dominated by the

assumption that the market represents the most efficient mechanism for allocating economic

55

resources in society. Economics does not assume a role for government as a given. Economists see

the fundamental roles of government in as allowing market mechanisms to operate. Thus,

economics explains or justifies the role of government in terms of ‘market failure’(Freebairn, 1989).

O’Faircheallaigh et al. (1999), state that economic rationalists are particularly concerned with:

• technical efficiency – the relationship between the value of resources consumed in a specific

economic activity (inputs) and the value of resources that are produced as a result of that

activity (outputs) [Benefit-cost analysis]

• allocative efficiency – involves the degree to which the overall allocation of resources between

alternative, competing uses maximises total economic output.

Pusey (1996) points out that the theory of economic rationalism has had growing influence on

policy decisions in many liberal democracies, and a substantial impact on the policies of

governments towards their public sectors.

Influence of organisational theory and management on public sector

management

O’ Faircheallaigh et al. (1999: 43) point out that ‘organisational theory and management’ represents

not so much an established discipline but rather an approach that suggests that the design and

management of organisations is a subject that deserves separate and sustained study in its own

right.’ They go on to explain that its basic assumption is that that all organisations have features in

common, and that a systematic study of these features and issues can allow general conclusions

about the utility of particular organisational forms and processes in certain contexts. In this area of

study O’Faircheallaigh et al. (1999: 44) point out three major persistent themes:

• The need to re-structure organisations. Key issues include: the numbers of layers in an

organisational hierarchy, and where to locate and concentrate authority for decision-making and

how to allocate tasks in relation to organisational structure.

• The question of how large, complex organisations can be directed to achieve goals set by

relevant decision-makers.

• The promotion of strategic alliances and other collaborative mechanisms designed to exploit

complementarities and to achieve efficiencies through agreed divisions of labour (ibid).

56

Influence of political science on public sector management

O’Faircheallaigh et al. (1999: 39) state that political science (or politics) is concerned with the

‘good society’ and the allocation of resources. Public policy is concerned with ‘who gets what,

when and how’ (Lasswell, 1936). O’Faircheallaigh et al. explain how any change to the way in

which the public sector functions will be assessed, in part, on the basis of its impact on the capacity

of citizens to enjoy and exercise rights. Thus the relationship between inputs and outputs (technical

efficiency) is not the only criterion for gauging the desirability of public sector change. In modern

society it is impossible for the vast majority of citizens to participate directly in collective decision-

making. Thus, a critical issue becomes the mechanisms through which those who do participate

directly (i.e, elected politicians) are held accountable for their actions (O’Faircheallaigh et al.,

1999).

O’Faircheallaigh et al. (1999: 43) state that political scientists are likely to view the ‘performance’

of the public sector in ways that are both complex and not easily amenable to quantification.

Assessments of performance are likely to involve trade-offs. Efficiency is important, but so is

respect for citizens’ rights and the promotion of an effective accountability mechanism

(O’Faircheallaigh et al., 1999: 43).

Influences of marketing on public sector management

O’Faircheallaigh et al. (1999: 46) describes marketing as an activity that ‘has traditionally been

associated with the distribution and promotion of commodities, and more recently of services, in the

private sector’. They suggest that the application of a marketing approach to the public sector

involves significant ambiguity. In the private sector consumers exercise their power by their choice

to purchase or not, and thus their power is contingent on consumer capacity to pay rather than on

the intrinsic needs or rights of citizens to basic services. Thus arises the danger of allocation of

resources based on capacity to pay. On the other hand, application of a marketing perspective could

be seen as a way of reasserting the primacy of client interests as opposed to bureaucratic or

professional interests (O’Faircheallaigh et al., 1999).

57

3.5 A Case Study of the Agriculture Division in the

Victorian Department of Natural Resources and

Environment

As each state of Australia has a unique history of public sector management and provision of

extension, a case study of the reforms in Victoria is provided, rather than attempting to generalise.

This includes a brief examination of the State government policy, followed by a more in-depth

analysis of the new public management (NPM) reforms on the Agriculture Division of the Victorian

Department of Natural Resources and Environment (NRE). The implications these reforms pose to

the practice of program evaluation are also considered.

The field work for this thesis was conducted from 1997-1999, under a Liberal state government.

Therefore the emphasis will be on reforms in NPM that were largely brought about between 1992

and 1998 when Victoria was under this Liberal state Government, with Jeff Kennett being the state

Premier. The most comprehensive statement of the Kennett model was spelt out in the 1993

Management Improvement Initiative 1993 (cited by Alford and O’Neill, 1994: 4):

• Focus on clear responsibility and accountability for results – ie, emphasis on outputs and

outcomes

• Empowering consumers – fostering consumer choice

• Minimising government bureaucracy – focusing departments of core functions

• Preference for market mechanisms – privatisation, contracting out, and competition policy

• Professional and business-like management of public agencies.

This statement is very much in line with the rhetoric of New Public Management that appears to be

influencing public sector management in most OECD countries.

The following section focuses on the impact of new public management reforms on one government

department: the Agriculture Division, the Victorian Department of Natural Resources and

Environment (NRE), and specifically on the provision of agricultural extension. The Agriculture

Division is the part of the Department of NRE that is responsible for service delivery to agricultural

industries and includes the state’s purchasers and providers. The range of services includes

extension, research and development, policy, regulation and trade development.

58

Influences of economics on the Agriculture Division

In July 1996 the purchaser-provider model was introduced for publicly-funded agricultural

extension in Victoria. Consequently, for the latter part of 1990s, the language used became that of

purchasers and providers, contracts and tenders. The major issues that related to the move towards

the semi-privatisation of the provision of public-sector services included the increasing pressure on

funding and the purchaser-provider split.

Increasing pressure to justify funding

Watson et al. (1992), in a Review of Field Based Service, recommended that extension in the

Victorian Department of Natural Resources and Environment should concentrate on areas of market

failure where the private sector was unwilling or unable to provide services to farmers. This

represents a departure from the worldwide tradition of governments taking the financial

responsibility for providing agricultural extension services. The prime reasons were to increase

agricultural production or to improve the efficiency of agricultural production (Lees, 1990). Both

reasons were concerned with increasing output to provide more food, fuel, and fibre to meet

growing domestic or export demand. A Victorian state departmental corporate plan gave the

overall rationale for the continuation of government extension services as to support increased

income from primary production within sustainable agricultural systems (Watson et al., 1992).

However, the relative importance of agriculture in the economic growth of industrialised countries

has markedly declined (Blank, 1999). A system of user pays is frequently held up as the most

effective way to overcome the decline in extension budgets. This change to a user-pays system has

already occurred for the provision of agricultural extension in Tasmania and New Zealand (Marsh

and Pannell, 1997). However, in Victoria the prospect of moving to a user-pays system has

somewhat diminished (McDonald, B. pers. comm., 2000). Nevertheless, public funding for

agricultural extension is frequently questioned and projects must meet increasingly high standards

of accountability and quality to prove their worth. Well-designed evaluation strategies that include

some formative component of evaluation are seen as an essential basis for planning, implementing

and improving high quality agricultural extension projects, and hence assuring public benefits from

research and development expenditure. The pressure is also increasing to prove that they have

produced outcomes that deliver public benefits. Thus, both formative and summative evaluation of

extension projects is critical to justify continued public funding.

59

The purchaser-provider split

In Victoria a version of the purchaser-provider model was introduced in July 1997 for government-

provided agricultural extension. With the separated roles of purchaser and provider, the purchasers

specify the services required, and assess their provision by the providers. In simplistic terms, the

purchaser role is responsible for investing resources wisely, and demonstrating the value of the

investment. The provider role is to provide the service contracted and to assist with demonstrating

the value of the service. In Victoria, the purchaser-provider model has not been implemented as

‘purely’ as in other states such as Western Australia. In Victoria, certain providers are nominated as

‘preferred providers’ and this includes the providers internal to NRE (Shaw, H. pers. comm., 2000).

Elsewhere, purchasers have more freedom to purchase across a wider variety of providers.

According to internal documents (NRE, 1997), the purchaser group in NRE aims to increase

impact, decrease overhead costs and to improve evaluation, among other things. The purchaser

group also has a strong outcomes focus. These outcomes are broadly defined at the outset of a

project, then later in the project development cycle they are defined more specifically, i.e.; to be

specific, measurable, accountable, realistic and timebound (SMART). The projects encouraged by

this group are to be focused on the achievement of outcomes and should have a built-in evaluation

strategy. To obtain tenders, providers will have to move towards projects with a built-in evaluation

component (NRE, 1997). It seems the development of evaluation plans conducted largely by

internal evaluators is being favoured.

The purchasers are expected to define outcomes for industry. From 1997 onwards project proposals

without clearly defined outcomes and built-in evaluation strategies are less likely to obtain funding

(McDonald, B. pers, comm., 1999). In addition, evaluation information is increasingly being seen as

important in terms of communication between purchasers and providers (McDonald, B. pers.

comm., 1999). Under the new model, purchasers and providers no longer work alongside each

other in a manager-employee relationship. Added to this is an increased number of investors and

co-deliverers, which means that projects have become more complex to manage.

Another interesting dilemma arises out of this new relationship. While the purchasers may not be

actively involved in delivering services, the farmer-clients will blame the government (and it is the

purchasers who receive this feedback) if services are not considered up to scratch. Thus evaluation

and accountability become important in the relationship between the purchasers and the providers.

60

Influence of organisational theory and management on the Agriculture Division

The Agriculture Division in Victoria has had up to 2,500 projects operating at one time. In an

internal report in 1997, a strategic objective was to reduce this to a portfolio of 50 projects by 2001.

Since then, many of the smaller projects have been amalgamated and internal documents state that

funding will only being given on the proviso that the new projects feature an investment of greater

than $1m pa and have quantified, timebound outcomes upon which all the project work is focussed

(NRE, 1997). This move to large projects is intended to increase administrative efficiency and to

focus more effort on achieving fewer, but more substantial outcomes. This is accompanied by a

change in the organisational level at which evaluation is being conducted. Previously, much

evaluation was done at the portfolio level and economic evaluation and performance indicators

were applied across an entire industry. Lately, with the move to smaller government, evaluation

responsibilities have been partially devolved to those who manage these mega-projects.

Promotion of strategic alliances

One of the Agriculture Division’s strategic objectives is to double the level of non-Victorian

Government investment in Primary Industries brokered projects (NRE, 1997). The idea of this is to

promote strategic alliances with other investors, such as the Rural Industry Research Corporations

(RIRCs) and other purchasers of NRE services, interstate agricultural departments, universities,

CSIRO, and other Commonwealth agencies, and private sector investors. To improve their ability

to create strategic alliances, the purchasers involve all the co-investors in all stages of the project

management (NRE, 1997).

Promotion of evaluation as organisational learning

In addition to its accountability purposes, evaluation under the new reforms, is also expected to

contribute to continuous improvement of extension activities (NRE, 1997). Rist (1997) suggests that

with the move to smaller government, public sector managers are expected to learn from their

mistakes and design ways of constantly improving their performance. Public sector managers are

now expected to be flexible, responsive, cost effective and innovative. It seems that the devolution

of evaluation to the project level is expected to result in increased utilisation of evaluation findings,

i.e., that projects will use the findings to improve their practice.

61

Implications of restructure for extension evaluation

As a result of the restructuring and increased collaboration, the new organisational structure under

which the Agriculture Division’s extension projects operate is complex. These mega-projects may

have several purchasers, influencers, and co-providers. For example, the Target 10 Dairy Extension

Project has four funding organisations, five investment managers and four providing organisations.

Added to this, other key stakeholders (farmer groups, environmental groups and industry bodies)

have voice, or representation in all the categories (McDonald, B. pers. comm., 1999). The various

organisations involved have distinct organisational values and cultures; as exemplified by the

disparate cultures of private agribusiness, university, and the public sector. The diversity of the

organisations the investors represent (not to mention the personal values any additional investors

bring), means that there is also a great range of values to manage while evaluating these mega-

projects. For example, the views of a farmers’ union representative on a good project outcome may

be quite different from those of a public sector investor.

In addition to diversity in terms of investor expectations of project outcomes and impacts, the

various purchasers may also have different evaluation information requirements. There are also a

whole range of other stakeholders, such as the farmers themselves, whose needs may need to be

addressed in an evaluation. Toulemonde et al. (1998) suggest that managing diverse values of

stakeholders in complex projects can be problematic. They point out that ‘partners that are

politically independent, have legitimately differing objectives, and are not accountable in the same

way to citizen partnerships, tend to complicate evaluation work at each stage’ (ibid: 171).

Taking this organisational complexity into account, it is vitally important that time is allocated for

the various stakeholders to enter into a meaningful dialogue about what is happening in the field,

and whether these experiences represent the sort of outcomes that are desirable. Project managers

are nowadays expected to provide evidence that they are capable of reflective practice and

managing organisations in a context of rapid change.

Influence of political science on the Agriculture Division

Internal documents of the Agriculture Division in Victoria bear witness to that fact that it is not

considered sufficient to produce only a cost-benefit analysis in evaluating projects and there are

frequent calls for identifying ‘who gets what when and how’ (NRE, 1997). There is also an

emerging disquiet about the limitations with the current performance measurement systems and

62

their capacity to deliver useful information (Winston, 1997; McDonald and Kefford, 1998). The

result of this is that purchasers are demanding that the new project proposals include evaluation

strategies that accurately ascertain:

• The return on investment in programs and projects (economic, tangible, and intangible)

• the requirements of customers and how well these have been delivered

• the practices and procedures for continuous improvement in project delivery.

Implications of politics for extension evaluation

Evaluation strategies will need to incorporate ways of determining whether both tangible and

intangible outcomes have been achieved, as well as describing the array of unexpected impacts

caused by the project interventions. These requirements have led to a move away from reliance on

economic analysis and simplistic performance indicators, towards the use of mixed methods

(including qualitative approaches) to capture the array of impacts, some of which may not be

quantifiable.

Marketing influences on extension

The Agriculture Division is moving towards a philosophy of market-driven service provision. They

aim to respond to the expressed needs of their client market and have set up formal links with

industry in an attempt to ensure that industry needs are met. These partnerships have a direct input

onto the strategic planning process used to assess industry priorities. As mentioned earlier, state

funding of extension projects is contingent on co-funding from the respective industry body. For

example, the Target 10 Dairy Extension Project receives 60% of funding from NRE and 40% from

the Dairy Research and Development Co-operation, which is an industry body (funded by levies on

milk).

Implications of marketing for extension evaluation

Under the new environment of extension, evaluation needs to provide sound information about the

customers – both co-investors and service users, their respective requirements and how well these

are met. This means there is a requirement for needs analysis, market segmentation and customer

satisfaction assessment (McDonald and Kefford, 1998).

63

Summary of changes in the Agriculture Division

The new public management reforms in the Agriculture Division, NRE, Vicotira, are summarised as

follows.

a) The appropriateness of continued public funding for extension was questioned extensively in the

late 1980s and early 1990s. Despite the fact that funding has stablilised recently, there is constant

pressure to justify public funds for extension. Evaluation is being seen as a way to improve the

efficiency of extension and to verify that it is returning benefits from government expenditure on

research and development.

b) With the purchaser-provider split, accountability and communication of evaluation findings

between the purchaser and providers has become more important. As a result of restructuring and

increased collaboration, extension projects have become bigger and more complex in structure. The

various organisations involved have distinct organisational values and cultures. There is diversity

in terms of investor expectations of project outcomes and impacts and the various stakeholders have

different evaluation information requirements. Evaluation approaches are needed which can deal

with complexity and value pluralism.

c) The organisational level at which evaluation is being conducted and directed has changed.

Previously evaluation was mainly economic and performance indicator-based, and was done at the

portfolio level. Other sporadic non-economic evaluation was directed from the central portfolio

level, which consisted largely of ‘expert review’ (where the views of key stakeholders or people of

influence were sought) (McDonald, B. pers. comm., 1999). In the late 1990s a trend began for non-

economic evaluation to be planned and directed by those who manage the mega-projects. The trend

is towards internal evaluation, with facilitation by evaluation experts, and some contracting out.

d) At the project level, there is demand for formative evaluation to enable organisational learning.

The intention is to enable projects to be flexible, responsive, and cost effective. At the project level,

summative evaluation is being demanded, to provide evidence of project outcomes and impact.

Evaluators are being asked to determine whether both tangible and intangible outcomes have been

achieved, describe the array of unexpected impacts caused by interventions and determine the

extent to which the farmer-clients perceive that needs have been addressed.

64

Contradictions in policies

New Public Management reforms have been influenced by economics, political science,

organisational theory and management and marketing. These reforms have occurred widely in The

Agriculture Division. Considering these recent changes together there are a number of striking

contradictions in policy, including:

• The Competition Policy (to increase public sector efficiency along private sector lines), versus

the policy of encouraging collaboration (with the move to smaller government, the public sector

providers are being encouraged to collaborate with industry to achieve outcomes).

• The policy of emphasising specific, measurable, timebound outcomes versus the call for

evaluation that captures intangible outcomes, and to capture change that cannot be quantified.

• Policies that aim to benefit the economy, rather than the individual farmer, yet a heavy focus on

(individual) farmer satisfaction.

• Delegation of delivery to providers, yet public accountability for impact resides with the

purchasers.

Given these contradictions, I suggest that the new model of public-sector management, as applied in

the Department of NRE in Victoria, has not yet stabilised. There appear to be serious limitations in

the application of the private sector management approaches, that are strongly reliant on the market

mechanisms, for public sector management. Pollitt (1997) suggests that most of the reforms are

more a matter of faith than of carefully evaluated policy and proven achievement. Project

evaluation seems to be wedged in the midst of all this chaos, and is increasingly being looked to as

the vehicle to solve problems created through changes in policy; especially concerning the problems

that complex project structures and new organisational relationships pose. Rist (1997) suggests that

the expectations for what evaluation can accomplish are growing, almost in direct proportion to the

pressures being felt by those in the public sector.

65

3.6 Conclusion

In Australia, since the introduction of the NPM reforms, the demand for evaluation appears to have

increased both in the extent of evaluation activity and in the diversity of the forms of evaluation.

Overall, evaluators are being asked to capture more information, for a larger audience. Since the

purchaser-provider split, purchasers are demanding more information to ensure that they are

investing wisely, as they are accountable both to the state and to the farmer-clients for achieving

project outcomes. With the move to fewer, larger projects with high levels of collaboration there are

more audiences for the evaluations of these projects than in the past. These audiences have an

increasingly diverse range of expectations in terms of evaluation findings. This means that

evaluation strategies will have to be negotiated carefully with the relevant stakeholders.

Despite the fact that the task of evaluation is becoming increasingly complex, in practice it seems

that a greater emphasis is being placed on developing an internal evaluation capacity amongst staff

at the project level. This internally driven evaluation is intended to satisfy accountability

requirements, to promote organisational learning and to provide credible information for

communication between the now separated purchasers and providers.

There are significant changes in the philosophy that underpins the conduct of extension. Evaluating

participatory and process-orientated projects is complex as the outputs of such projects tend to be

diverse, unpredictable and context-specific. Full participation in extension projects implies that

participants have control over intended outcomes, not just the process of achieving these outcomes.

Conventional, objectives-based approaches are inadequate to capture the range of unpredicted and

unanticipated outcomes that these projects produce. External evaluation against pre-defined

indicators of success would be incongruent with the philosophy of these new paradigm approaches.

To evaluate new, participatory, group-oriented approaches to extension, evaluation is required that

is participatory, process focused, inductive and largely qualitative.

I suggest that evaluation can be tailored to meet some of the challenges posed by new public

management reforms and by the changes in the culture of agricultural extension itself. A way

forward could involve a combination of formative evaluation, and summative evaluation. The

formative evaluation should be participatory yet facilitated by skilled evaluators and aimed at

organisational learning and helping communication between the array of stakeholders, including the

66

purchasers. It could be focused more on process than outcomes. One way of doing this would be

through a participatory monitoring system (such as is practised in the development projects in lower

income countries), in order to allow continuous learning, and involvement of various stakeholders.

I suggest that the summative evaluation component be designed to meet the need for credible

evaluation with high face validity to satisfy accountability purposes. This summative component

should go beyond simply asking whether a project works or not, and also determine for whom it

works in what situation and why (some of the theory-driven approaches as described in Chapter 2

could be appropriate). In the following chapter an examination of current evaluation practice in

extension will be presented in order to determine whether approaches to extension evaluation exist

that can meet these needs.

67

CHAPTER 4

REVIEW AND DESCRIPTION OF CURRENT PRACTICE OF

EVALUATION IN AGRICULTURAL EXTENSION

In this chapter the findings of a review of evaluation in agricultural extension in Australia are

presented. Fifty evaluation documents were categorised and analysed and using Owen’s

(1993) meta-model that is based on distinct purposes of evaluation. Nine case studies are

presented to illustrate the range of forms and approaches in practice. Where possible, the

evaluation studies were examined against the meta-evaluation criteria layed out in Chapter 2.

I found that there is very little monitoring of projects, and a tendency to use a handful of

familiar methods and approaches. Most of the studies were limited to a summative

assessment of project impact, and there was a pre-dominance of objectives-testing

approaches. I argue that while there is a range of forms and approaches to evaluation in

practice, the current suite of approaches is inadequate to meet all demands for the evaluation

of extension in Australia. In particular, no evaluation study was found that facilitated on-

going communication between multiple project stakeholders.

4.1 Introduction

I have argued that approaches to evaluation should meet certain premises for good evaluation and

be grounded in knowledge of context-specific evaluation needs (Chapter 2). An analysis of the

changing nature of extension organisations revealed an increasing demand for evaluation that can

deal with participatory, group-based extension models, provide accountability and promote

communication and learning between multiple stakeholders (Chapter 3). I suggested the evaluation

needs of large extension projects are varied and can only be met through a ‘basket of choices’ of

approaches to evaluation. Even for one project, an evaluation bricolage consisting of a number of

approaches and methods may be required to meet all the evaluation demands. In order to address

some of the challenges facing extension evaluation, this thesis aims to develop a model of

68

evaluation that can complement the current range of evaluation praxis. Therefore, before selecting

an evaluation model to adapt and pilot, an understanding of the current evaluation practice must be

gained.

This chapter addresses the question: What evaluation is already occurring in agricultural extension

in Australia? As a result of the scarcity of published literature, I conducted a review of mainly

unpublished reports of evaluation that were conducted on agricultural extension projects in

Australia over the last 10 years. This review was previously published in conjunction with several

other authors, (see Preface; Dart et al., 1998). The other authors helped collect the review

documents and write the final report (110 pages). I was solely responsible for reviewing and

analysing all the documents collected. I authored all the excepts that are presented in this chapter,

which I substantially revised since the Review of Evaluation in Agricultural Extension was

published.

The chapter begins with the background to, and justification for, the approach taken in reviewing

evaluation practice. I then present the findings of the review. The main method of presenting the

results of the analysis was to sort the sample of evaluation studies according to Owen’s (1993)

classification of evaluation forms. I then present some case studies to illustrate the different uses of

evaluation in extension. An attempt is also made to analyse the information in ways that may

provide insights into the range and value of evaluation studies being conducted within the field of

agricultural extension. After presenting the case studies I attempt to use the meta-evaluation

questions developed in Chapter 2 to review the 50 studies en masse.

4.2 Background to the Review

In 1997, at the start of this study, I conducted a search for published literature concerning evaluation

in the field of agricultural extension in Australia. I found very little published material. However,

published literature covers just one part of the ‘lived experience’ of applied evaluation: only the

information that authors choose to write about comes into the public domain; other potentially

valuable information often remains unrecorded and hidden. It could well be that practitioners do not

fully realise the value of their experiences for others in similar situations. The lack of information in

the public domain is also no doubt influenced by the confidential nature of many evaluation

findings. Another limiting factor may be the lack of fora for dissemination of this literature. There is

currently no Australasian journal of extension in which these findings and methodology might be

69

published.

The published literature existing tends to recommend the need for more evaluation and monitoring:

Lees (1990); McDonald and Kefford (1998); Dunn (1996); Woods et al. (1993); and Russell et al.

(1989). This finding was substantiated in the early stages of collecting material for this review. A

very common response to our request for information was that practitioners were just beginning to

learn about evaluation and had no written reports that they felt were good examples of evaluation to

include as case studies. This is reflected in the fact that of over 660 members of the Australasian

Evaluation Society, in 1997 only two listed agriculture as an interest (Australasian Evaluation

Society, 1997).

Indeed, there appears to be substantially less information available on extension evaluation

nowadays, than on evaluation of health or education programs where the field of evaluation is

constantly evolving to meet the needs of new programs and new ways of thinking. In 1997, I found

the existing reference information on extension evaluation to be largely out of date and inaccessible

to many extension agents. The current lack of literature concerning program evaluation in the field

of agricultural extension is perplexing to those who are aware that historically many important

contributions to program evaluation have been made from the field of agricultural extension (see

Section 2.2). Perhaps this can be accounted for by the character of agricultural extension itself.

Agricultural extension aims to change behaviour through the use of communication. Behaviour

change is commonly perceived to be difficult and complex to measure, especially in terms of

quantitative data. It is worth recalling that few agricultural extension workers have traditionally had

training in social science disciplines and they are likely to be unfamiliar with the range of

qualitative data collection methods.

However, since the completion of the review in 1998, the picture has been changing. Throughout

Australia training courses in program evaluation have multiplied. In Victoria a special project

named the ‘Evaluation Pilot’ was initiated in 1997 to facilitate over 50 major projects in the

Agriculture Division of NRE, to establish evaluation plans. This initiative involved intensive

training for staff in a range of evaluation issues, and individual support to projects. By May 2000,

over 50 staff from the Agriculture Division had been through this training, with another 50

participants expected to complete the training by 2001. At a national level the Rural Extension

Centre in Queensland and the University of Melbourne also offered increased provision of training

in evaluation specifically for extensionists. However, in 1998 when I completed the research for the

70

review of current practice, many of new and innovative approaches to evaluation were still in early

stages of implementation, and therefore were not included in this review.

4.3 Approach Taken

Documents were collected from over 80 evaluation studies from a wide range of sources, ranging

from rural extension centres, to universities and agriculture departments in various states, to private

consultants and individuals in the field of agricultural extension. These documents came from all

six states of Australia and were conducted between 1990 and 1997. This list grew to over 100

respondents from all states and some overseas specialists, through advice and further contacts

received from individuals and members of the Australasian Pacific Extension Network. Of these

documents, 50 were selected to provide the data for an analysis of evaluation in extension. The

remaining reports were excluded as they contained insufficient information, were primarily

evaluations of ‘one-off’ workshops, or were guides as to how to conduct a hypothetical evaluation.

This sample contained some inherent bias in that it comprised mainly documented evaluations and

did not include internal departmental monitoring and evaluation systems (although anecdotal

evidence suggested that little of this is occurring), or informal evaluations. Macro-level evaluations

were also excluded, and some innovative forms of evaluation research may have slipped through

the net because they are not always labeled as evaluation. It does however, provide a first glimpse

of the overall picture of approaches to evaluation in agricultural extension nationally.

Process used to review the evaluation studies

The 50 evaluation studies in the sample were characterised according to variables relating to (a) the

program, (b) the evaluation, and (c) the type of methods used. The evaluations were then classified

according to Owen’s meta-model (see below). Where possible, the meta-evaluation questions,

established in Chapter 2 were then used to critically examine the evaluation studies.

Initially, I intended to classify the case studies in terms of the normative models to which the

approaches related – such as the framework presented in Chapter 2 (experimental, objectives-

testing, judgemental, decision-management, intuitionist-pluralistic and theory-driven approaches).

However, on reading the evaluation documents, it became clear that almost none of the documents

explicitly referred to the underlying approach or model of evaluation. For example, only two

71

theory-guided approaches to evaluation were found, no overtly pluralistic-intuitionist approaches

were referred to, and only one document referred to the use of a decision-management approach

(Utilization Focused Evaluation). Most of the evaluation studies did not refer to any evaluation

literature, or popular models of program evaluation. Indeed, anecdotal evidence suggests that at the

time of the review, few people practising extension evaluation had any formal training in evaluation

and therefore they used the skills they have from other disciplines such as extension, or rural

sociology, to structure the evaluation. Some studies were based on social-science premises, others

on participatory rural appraisal approaches used in the practice of extension. For this reason, the

evaluation studies were classified and examined against a framework that differentiates between the

purposes of evaluation studies.

Owen’s meta-model

The conceptual meta-model of Owen (1993) was adopted to classify examples of evaluation studies

obtained from the field. This meta-model uses a relatively straightforward framework based on

purpose, and results in five forms of evaluation:

• Form 1 Impact evaluation

• Form 2 Evaluation in program management (monitoring)

• Form 3 Process evaluation

• Form 4 Design Evaluation (to clarify the program design)

• Form 5 Development evaluation (to develop new programs).

In essence, these forms relate to the different types of information that program staff might wish to

gain. Table 1 summarises the meta-model, and shows how each form is related to a specific

purpose. The forms of evaluation are also related to a timeframe within the program cycle.

72

Table 1 Owen’s five forms of program evaluation

Form of evaluation

Form 1

Impact evaluation

Form 2

Evaluation in program

management

Form 3

Process evaluation

Form 4

Design evaluation

Form 5

Evaluation for development

Ultimate purpose

Justification Accountability Improvement Clarification Synthesis

State of program

The program is settled

The program is settled

The program is at developmental stage

The program is at developmental stage

The program has not yet been implemented

Timing of evaluation

After program Runs throughout the program

During program implementation

During program implementation

Before program is implemented

Focus of evaluation

On outcomes/ delivery of the program

On delivery/ outcomes of the program

On delivery of the program

On the design of the program

On context of the program environment

Typical approach/ Methods used

Objectives-testing evaluation, needs-based evaluations, economic evaluations

Program monitoring, system evaluation, use of performance indicators over time

Implementation studies, action research, responsive evaluation, topical RRAs or PRAs, process evaluation

Evaluability assessment, program logic, accreditation

Needs assessment, review of best practice, exploratory RRA (rapid rural appraisal) ex-ante evaluation

Time span Program finish Throughout program

Program is settled

Program is developing

Program synthesis

Form 1 Form 2 Form 3 Form 4 Form 5

Source: adapted from Owen (1993: 22)

The domains of evaluation

Owen’s framework refers to five ‘forms’ of evaluation distinguished by the main purpose of the

evaluation. The extensive literature in program evaluation discusses various ‘models’ and ‘methods’

of evaluation. In Chapter 2, I presented a meta-model that categorised evaluation into six broad

normative approaches to evaluation. These various ‘domains’ in evaluation (form, framework,

model, and method) can be confusing, so an attempt is made here to explain the relationships

between various terms in common use in evaluation literature. Figure 4 illustrates the conceptual

relationship between the different domains of evaluation for the evaluation bricolage for a

hypothetical program. These terms were defined in Chapter 2.2.

73

Evaluation bricolage for one project Form 4 Normative Form 1

Evaluate program models Impact evaluation logic

m6 Form 2 m5

m1 m2 Monitor program

m4

m 3

m represents a method of collecting and analysing data

Figure 4 Example of an evaluation bricolage for an hypothetical program, showing three

domains

In Figure 4, the large ellipse represents the evaluation bricolage for an individual project. Within

this ellipse are smaller ellipses representing different ‘forms’ of evaluation. For this hypothetical

program, three forms of evaluation are required to meet three different evaluation purposes.

Conceptual models influence the choice of methods of data collection or creation and the forms of

evaluation adopted. It should be noted that one method of data collection or creation could be used

for more than one form of evaluation and that existing data may also be used. In this review the

evaluation studies were categorised according to form, rather than method, or model.

4.4 Description of the Evaluation Studies

Figure 5 shows the distribution of studies according to Owen’s (1993) forms. A difficulty

experienced in using this classification was that studies could often be placed in more than one

category, especially with regard to the time frame that Owen attached to the meta-model (see Table

1). The distribution of the studies according to a slightly modified Owen categorisation is shown in

in Figure 5.

Existing data

Existing data

74

Figure 5 Distribution of the evaluation studies according to Owen’s five forms

The distribution in Figure 5 shows that impact studies were the most common, but that process

evaluation (Form 3) was also frequently used. From information on the date of the studies there

appears to be a recent trend towards process evaluation, and also towards the use of evaluation for

formative rather than summative purposes in agricultural extension. This trend is congruent with the

rhetoric of many current extension funders who promote evaluation for learning, in addition to

evaluation for accountability purposes (for example the Agriculture Division in Victoria, as

discussed Section 3.5).

Process evaluation

37%

Evaluation for development

4%Evaluation for

design clarification

9%

Monitoring7%

Impact evaluation43%

75

4.5 Case Studies of Different Forms of Evaluation in

Agricultural Extension

In the following sections the five forms of evaluation from Owen’s meta-model (Owen 1993) are

outlined and illustrated with case studies from agricultural extension. Each of the case studies is

examined in terms of what it can offer the ‘basket of choices’ for extension evaluation, and

particularly what the approaches can contribute to the meeting demands for evaluation for large

extension projects (Chapter 3). The discussion in Chapter 3 revealed that evaluation is needed

which can:

• show the extent to which predetermined objectives have been achieved and satisfy

accountability requirements

• provide accountability for group-based models of extension that produce diverse, context-

specific outcomes

• include participatory process that foster learning of the participants and augment the

intervention effort

• be conducted as part of an internal evaluation capacity and hence foster organisational learning

• provide performance information for communication and between multiple stakeholders

• involve evaluation processes that foster dialogue between multiple stakeholders.

Form 1: Evaluation for impact assessment

Owen (1993: 23-24) describes ‘impact evaluation’ as evaluation that leads to a decision about the

worth of a program and which often has a strong summative emphasis. This type of evaluation is

generally carried out at the end of the program, or when a program is at a settled phase. The

principal foci of impact evaluations are to:

• understand the outcomes of the program

• justify program spending

• gain guidance about what to do next.

This was the most commonly found form of evaluation. Within this category of impact evaluation

in agricultural extension, several approaches were found. Each of these approaches is illustrated

76

with a case study under the following headings:

• cost benefit analysis for impact evaluation

• objectives-testing impact evaluation

• needs-based, participatory impact evaluation.

Cost benefit analysis for impact evaluation

Cost-benefit analysis (CBA) is a procedure for comparing alternative courses of action (or no

action) by reference to the net social benefits that they produce. A net social benefit refers to the

difference between social benefits and social costs. CBA is a method for organising information to

aid decisions about the allocation of financial resources (Department of Finance, 1991). The method

can be applied to specific projects or to larger programs as a whole. Its power as an analytical tool

rests in two main features:

• costs and benefits are each expressed (as far as possible) in money terms and hence are directly

comparable

• costs and benefits are valued in terms of the claims they make and the gains they provide to the

economy as a whole, so the perspective is a global one rather than one of any particular

individual, organisation or group.

CBA is widely used as a generic tool in evaluating the financial impact of agricultural programs. Its

application may also be ex-ante (ie. before implementation). However, cost-benefit analysis is often

used when a program has matured or ended, as part of the evaluation of impact or outcomes.

Practical constraints in using CBA concern the difficulties in assigning money values to the costs

and benefits of a project or program (Department of Finance, 1991). Ultimately CBA is a tool for

summative evaluation: to provide information for those who are to make judgements about whether

the program outcomes justify the expenditure.

The use of CBA is commonly criticised by agriculturalists on the grounds that unrealistic

assumptions are often used in estimating costs and returns, and because of the failure of the analyst

to take account of important issues. The findings of CBA often carry considerable weight in policy

decisions regarding the allocation of future funding. The fact that CBA reduces impact to dollar

values enables comparisons to be made across programs with very disparate outcomes. It has been

argued of late in the Agricultural Division, that agricultural extension professionals can often

77

provide quantitative information that can greatly improve the estimates and assumptions used by

economists in CBA (McDonald, B. pers. comm., 1999).

One of the main advantages of CBA is that it has high face validity with the Treasury and with the

people who make decisions about future funding. It offers a way of providing accountability for the

balance of benefits to costs. However, CBA has inherent biases and limitations, and should be seen

as one part of a comprehensive evaluation strategy rather than a stand-alone approach. For

example, it offers little to extension officers in terms of learning. It is an example of ‘black box’

evaluation, providing no information about why certain impacts have or have not occurred. It is

generally carried out by an external evaluator; this could be deemed as incommensurable with the

ethos of participatory projects, especially if it were to be the only evaluation tool used. It also

provides little in terms of facilitating communication and dialogue between project stakeholders. It

does not explore unanticipated outcomes, and is limited to considering impacts that can be

converted into dollar values.

Case Study 1 provides an example of CBA conducted on the Target 10 Dairy Extension Project in

Victoria.

Case Study 1 Cost–benefit analysis of the Target 10 dairy extension program

The program

Target 10 has the aim of enhancing the viability of the dairy industry through programs that profitably increase the consumption of pasture per hectare. Information from research and development on pasture utilisation is extended to farmers through courses, discussion groups, newsletters, target graphing, re-designed feed planning, comparative analysis, field days, focus farms and demonstrations and other media.

Goal of the evaluation

To estimate the economic benefits and to conduct a cost–benefit analysis of the expected results.

The evaluation and assumptions

This was a preliminary evaluation based on benchmark studies prior to the project launch in 1993. It does not consider all benefits from project activities, i.e., those not directly associated with the project aim. The first step in the cost–benefit analysis was to estimate the average amount of pasture consumed/ha and the amount of milk produced had this project not been implemented in 1991. These and other assumptions made are listed below:

level of pasture utilisation: 6.5 tonnes/ha (range 4-9 tonnes in Victoria) production from increased use: 7% increase in milk results from 10% increase in consumption milk production in Victoria: 511 514 litres/farm on 8469 farms (Q0)

78

levy and transport charges: 2.3 c/l (levy) and 2.7 c/l (transport) milk price received (P): 24.5 c/l – levies & transport charges = 19.56 c/l (P) average cost of production: 14 c/l before implementing Target 10 (AC0)increase in producer costs $2000/farm/yr required for implementation of Target 10

The maximum potential benefits resulting from Target 10 were calculated using the incremental profit method (Morrison 1993), i.e., from P = (P*Q1 – AC1*Q1) – ( P*Q0 – AC0*Q0) where Q1 (Q0 plus 7% increase in production) and AC1 are the Quantity of milk and Average cost with Target 10, respectively. The estimate of maximum potential per farm was $4728 and for Victoria was $40 038 500.

Method (the cost–benefit analysis)

The total nominal cost of the Target 10 Project was $9 198 000 from various sources. The spreadsheet model Appraisal (Appleyard 1996b) was used for the cost–benefit analysis. The benefit of the project is ‘accelerated adoption’. Appraisal was run twice: first using the adoption profile had Target 10 not occurred, and second using the adoption profile with Target 10. The Net Present Value (NPV) was calculated as the difference between the NPVs of those two scenarios. To calculate the BCR (benefit cost ratio), the difference between the two present values of the benefits was divided by the present value of the project’s costs.

Target 10 was designed to accelerate the rate of adoption of favourable innovations related to pasture utilisation (years from information being available to uptake by farmers). In addition it was expected to increase the level of adoption (percent of farmers). It was assumed that without the project, maximum adoption of the relevant practices would be 32%. In 1997 40% of Victorian farmers attended Target 10 training courses across Victoria. Some further assumptions and the results of applying the Appraisal model are summarised in the table below: Assumptions Without Target 10 With Target 10 Year adoption begins 8 2 Maximum adoption 32 40 Years to obtain maximum adoption 15 7 Maximum potential benefit 40 038 519 40 038 519 Cost benefit analysis Present value benefits ($) 57 404 678a 120 084 711b

Present value of costs ($) 6 763 653c

Net present value ($) (b-c)-a 62 680 033 Benefit-cost ratio 10.27 IRR (%) 140.15

A sensitivity analysis was conducted for a number of possible adoption profiles. The year adoption begins and the maximum level of adoption without Target 10 are of particular interest because these contain the greatest amount of uncertainty. The year to reach maximum adoption was held constant and the results are shown in Figure 6:

79

-100

1020304050607080

2 3 4 5 6 7 8 9 10 11 12

NPV

$m

Year adoption begins

Max adoption 32% Max adoption 36% Max adoption 40%

Figure 6 Graph of net present value at different adoption rates

The graph shows that the year in which adoption would have begun without Target 10 has a large effect on the project’s outcome. The longer it would have taken for adoption to occur, the greater the benefit from extension work. The greater the acceleration due to Target 10, the greater the benefits. The graph also shows that the increase in maximum adoption resulting from Target 10 also affects the size of the benefits. A break-even analysis was conducted to work out the maximum costs that the producer could incur from implementing Target 10, before incurring a loss. Assuming a 7% increase in milk production from a 10% increase in pasture utilisation, this was found to be $5000 per farm.

Use of findings

Using the assumptions stated, the analysis indicates that extension has sped up adoption by six years, resulting in a net benefit of $62 million to the Victorian dairy industry. This preliminary analysis helped to validate project plans, and will be followed up by other evaluations by means of surveys of change in pasture use and farm profitability (Boomsma et al., 1996).

Sources: Boomsma et al. (1996); Appleyard (1996a); Appleyard (1996b) and Morrison, (1993).

Objectives-testing impact evaluation

Objectives-based evaluations judge whether the stated goals or objectives of a program have been

achieved. Tyler (1967) was amongst the first to develop and use goals-based evaluation. In this

approach goals are taken as given, and decisions about the success of a program are deduced from

whether the program has met its goals. According to Owen (1993) the main tasks in planning a

goal-based evaluation are to:

• determine the ‘real’ goals of the program

• decide how to determine whether the program has led to the attainment its of goals.

80

A standard procedure in objectives-based evaluation is to develop measures of goals that have

strong face validity. However, sometimes it is necessary to use substitute measures or indexes that

stand in the place of preferred or ideal measures. An example of a goal-based evaluation is given in

Case Study 2. In this case study, the goals of the Landcare program was taken as ‘given’, and the

evaluation essentially tested whether farmers in Landcare groups had greater awareness of

environmental issues than farmers who were not members of Landcare groups. This type of

evaluation has strong credibility for evaluation audiences in terms of proving that a program has

met some of its defined objectives and may be an important part of an evaluation bricolage of a

program.

However, it does not meet all evaluation needs and it has inherent bias and limitations. Firstly, it

does not investigate the validity of the objectives of the Landcare program. Landcare is a group-

based project, and each group may have individual objectives. This evaluation does not attempt to

explore these context-specific outcomes. The evaluation was externally conducted, and therefore,

would not foster organisational learning that a more internally derived process may have

precipitated. This evaluation is not participatory, and could even have encountered some hostility

by the Landcare group members, who could feel that the evaluation is being done to them. It does

not utilise evaluation processes that would foster dialogue between multiple stakeholders. The

evaluation design, being objectives-focused, does not investigate unanticipated positive or negative

outcomes of the program. It focuses on changes in attitude and knowledge, but does not investigate

changes in behaviour – which are after all the main purpose of extension. As in all quasi-

experimental designs, there is no foolproof way of proving that the two groups (those already in

Landcare groups and those not in Landcare groups) are in fact really representative of the same

population. Despite all these limitations the approach described in this case study was rigourous in

attempting to determine the extent to which valued objectives have been achieved, and may well

have served its purpose well. Evaluation of this type is a valuable contribution to the ‘basket of

choices’ for extension evaluation, but is not advocated as a stand-alone approach.

81

Case Study 2 Landcare: does it make a difference?

The program The Landcare program emerged in the state of Victoria during 1986 and has been spread throughout Australia as a model for effective community action to manage land degradation and assist the move to more sustainable land use. The evaluation described here was carried out in 1993, at a time when the Landcare program in Victoria could be described as in a settled state, with many groups having been operating for six years. This evaluation forms part of a wider Landcare evaluation undertaken within the Johnstone Centre in Wagga Wagga. This program is based on the assumption that Landcare group actions will facilitate the process of community development and thereby produce more aware, informed, skilled and adaptive resource managers with a stronger stewardship/land ethic, and result in the adoption of more sustainable natural resource management practices.

Goal of the evaluation To assess the effectiveness of Landcare groups by investigating the work of groups and their impact upon participants’ behaviour and on other rural landholders. Specifically, the study aimed to determine whether Landcare participants and non-participants were significantly different in terms of key social and farming variables and whether Landcare participation made a significant difference to landholders’ awareness of issues, level of knowledge of key resource management, and their stewardship ethic.

Form of evaluation The overall purpose was to investigate some aspects of the overall program aims. Owen’s Form 1, ‘Impact Evaluation’, best covers this example. This evaluation made summative judgements as to whether the program had met intermediate goals of the Landcare program, so took an objectives-testing approach. This evaluation attempted to measure the degree to which the goals (or substitute goals) have been met by program activities.

Method of evaluation A regional case study was carried out. The study comprised a comparison between Landcare respondents with non-Landcare respondents, and between respondents in Landcare areas with respondents in areas where Landcare groups did not exist. In April 1993 a 16 page survey was mailed to all land managers of rural properties in 12 catchment areas (nine Landcare areas and three non-Landcare areas) of Northern Victoria. Surveys were addressed to one listed owner per rural property, 30% of who were women. The researcher’s knowledge of the area and links with Landcare group there played a critical factor in the selection of the region. All groups surveyed had been operating for more than two years. The surveys were pre-tested using two focus group of Landcare participants and non-participants at separate locations. The individuals were contacted by telephone and given a brief overview of the project and invited to the pre-test focus group. The pre-test participants were mailed a survey before the focus group took place. Important revisions of the survey were made as a result of recommendations following the discussions at both meetings.

The questionnaire was a small (B5) booklet with a distinctive, authoritative cover and was posted with a covering letter and stamped return envelope. A reminder / thank you letter was posted after 8-10 days and a second mailout occurred after six weeks. Considerable effort appears to have been put into selecting appropriate topics to include in the survey. Factors such as property size, hours worked off-farm, length of experience as a farmer, membership of other groups, age of respondent and school education were correlated against participation in Landcare.

Use of findings The results were quantitatively analysed using chi-square and multivariate analysis. The statistical findings were then represented in tables with accompanying text to explain the findings. These findings were presented and published in a booklet.

Source: Summarised from a report by Allan Curtis and Terry De Lacy (1994)

82

Needs-based impact evaluation

Scriven (1967) offered a different perspective towards impact evaluation. He suggested that in some

circumstances objectives-testing evaluations have limitations. His argument was that testing the

goal (objectives) of a program does not tell the worth of that program for society in general, in that

it does not assess the goals themselves. In this thesis the term ‘needs-based’ evaluation will be used

to refer to evaluation where the orientation of the evaluation is directed toward the needs of society

or the stakeholders rather than a judgment of whether the given goals were reached. For example, if

the goal of a hypothetical program were to produce a million tonnes of beef as cheaply as possible,

a strictly goal-based program would work out whether this goal (and perhaps intermediate goals)

had been achieved. A needs-based evaluation, in addition to investigating whether the given project

goals have been met, might also look at whether program met the needs of the farmer’s and what

impact the program had on the environment, on the community and so forth.

The choice between a ‘needs-based’ approach to evaluation and a ‘goals-based’ approach should be

made with regard to the underlying purpose of the evaluation. Patton (1997) suggests that to be an

effective evaluator one needs to be able to evaluate with or without goals. Case Study 2, involving

the West Hume Landcare Group, gives an example of a needs-based, impact evaluation. It presents

quite a contrast to the previous case study and is much more participatory in orientation. Like the

intuitionist-pluralistic approaches discussed in Section 2.4, the evaluation in this case study attempts

to draw the farmers into dialogue with regard to what they consider to be important, rather than

testing ‘given-objectives’ as in Case Study 1.

This approach does appear to be able to meet some of the challenges discussed in Chapter 3. It

could go someway to providing accountability for group-based models of extension such as

Landcare that produce diverse, context-specific outcomes. Being a participatory approach it may

well be more commensurable with this type of project and foster learning of the program

participants. Again it does provide processes that can foster dialogue between multiple stakeholders.

It also goes some way to meeting demands for summative evaluation for accountability. It uses a

novel mix of approaches, including economic analysis. I applauded the evaluators for attempting to

address many of the challenges facing large participatory extension programs. However, it does not

provide any on-going information for learning, and was conducted while the program was in a

settled phase of development. It appears also to have been a highly resource intensive evaluation,

that might not be practical in every situation.

83

Case Study 3 : A case study undertaken with the West Hume Landcare Group as part of an international research project on Participatory Watershed Development

The program West Hume Landcare group was chosen to be one of two case-study evaluations of the National Landcare program. The group was established in 1989. There are 160 farms in the area and 85 families are group members. The main land degradation issues are rising water tables and salinity associated with a massive change in the natural vegetation, soil acidity, soil structural and nutrient decline. The evaluation was carried out when the group was well established. Nationally, there has been growing concern about how to evaluate local Landcare activities in a way that is useful to local groups, but which can also provide information to assist broader-scale policy development and help give a national perspective on Landcare achievements. This concern led to LWRRDC providing additional funding for a case study and to support the active involvement of a National Landcare facilitator.

Goal of the evaluation The overall goal of the evaluation was to appraise the success of Landcare, its shortfalls and future opportunities. More specifically the goals were to:

• enable Landcare group members to review the achievements, shortfalls and future opportunities for the group

• provide analysis of implications for Landcare policy • understand the economic outcomes of the group effort • provide an evaluation case study on Landcare to be presented at an international conference • provide a draft manual report for LWRRDC on how other groups could undertake a study such

as this, using a PRA (participatory rural appraisal) approach.

Form of evaluation This evaluation approach is ‘needs-based’ as opposed to ‘goals-based’ in that it attempts to evaluate against the needs of the community rather than against predefined goals. It is most similar to Owen’s Form 1, impact evaluation. However, it has both formative and summative elements: while it judges the performance of the project against community needs (summative), it also offers recommendations for the future (formative). This evaluation uses mixed methods ranging from participatory self-evaluation to economic analysis by external consultants.

Actors

External evaluators were used but their role was that of facilitation and analysis. There was a high level of community involvement in the evaluation. The audience was both internal (group and Landcare program) and external (IIED, LWRRDC).

Methods of evaluation

Four methodological approaches were used:

1. Qualitative data were gathered by means of a PRA. PRA is a method for rapidly gathering key information about a rural situation by drawing on local knowledge and integrating it with the knowledge and experience of professionals from a range of different disciplinary areas. The PRA was conducted for a period of seven days in 1994. Team members came from a variety of disciplinary areas. PRA techniques included semi-structured interviews, group histories, questionnaires, preliminary analysis of cost–benefit of some common land conservation practices, and an economic study of two local farms and two hypothetical best management practice farms on whether farmers could afford Landcare.

84

2. Some quantitative data were collected using a structured questionnaire, also during the PRA exercise, using semi-structured interviews.

3. Secondary information in the form of reports and previous studies were also examined and incorporated into the exercise.

4. Economic evaluation was carried out by external consultants prior to the PRA exercise. The purpose of that study was to determine whether farmers could afford Landcare. A detailed financial analysis was conducted of four typical farms of different sizes and enterprise mixes, and two hypothetical ‘best practice’ farms. Data for the analysis of the four farms were obtained by interview with four selected landholders.

PRA: during the day, team members conducted focus groups with a range of stakeholder groups. In addition, members travelled to farms and interviewed individual farm families. Groups included Landcare members, non-members, senior members of the community, a Landcare committee, and a women’s’ group. Each evening the team presented and analysed the findings during group discussions. At the end of the week the first results of the analysis were presented to the community and team during a dinner.

Use of findings

As well as being written as a report, this case study was presented at an international workshop on participatory watershed development in Bangalore, India, amongst 23 case studies from different countries. The West Hume case study was presented by the participating farmers and Landcare group members and was especially appreciated by people at the conference.

Source: Summarised from Woodhill et al. (1994)

Form 2: Monitoring and evaluation for program management

Monitoring and evaluation (M&E) for program management (based on Owen’s Form 2) is designed

to run throughout the life of a project and is generally set up during the design stage. This

distinguishes it from other forms of evaluation, which are more discrete studies. The main focus of

this form of evaluation is to provide continuous feedback to the program management information

system. While collecting material for the review several organisations reported that they were

initiating systematic monitoring systems, but I found little evidence of programs/projects that had

systematic, formalised, established monitoring systems at that time. This finding is confirmed by

the conclusion of Woods et al. (1993) that there was little formal M&E in Australia for the

management of extension.

In this section, two approaches to monitoring and evaluation are presented:

• planning-based approaches to monitoring using performance indicators

• evolutionary approaches to monitoring, that do not employ performance indicators.

85

M&E for program management is conventionally associated with input-output monitoring which

would be represented by levels 1 to 3 of Bennett’s Hierarchy (Bennett, 1977) (see Table 2).

Bennett’s Hierarchy is a widely used framework in extension evaluation, and is a postulated

outcome hierarchy for extension. Bennett describes a simplified chain of events assumed to

characterise most programs in extension. This chain of events is depicted as a hierarchy of

objectives and evidence for program evaluation. Bennett presents this hierarchy as a framework for

a goal-based approach to evaluation. The essence of this is that the more nearly the objectives of a

program are reached, the more positive the judgement of the program. The lowest level of Bennett’s

Hierarchy concerns the activities of implementing the program (such as contacting farmers, holding

meetings), while the highest level (Level 7 in Table 2) involves quite changes in farmer behaviour.

Bennett explains that it becomes increasingly difficult to attribute the changes to project activities as

the hierarchy is ascended.

Table 2 Bennett’s Hierarchy

Level Bennett’s Description Type of evaluation Level 7 Consequences for the target group Evaluation & monitoring relating to

social / economic impact Level 6 Behavioural changes in the target group

(Practice change, sometimes indicated by change in quality of products)

Evaluation & monitoring relating to behaviour change (also impact monitoring)

Level 5 Change in knowledge, attitude, skills, motivation and group norms

Evaluation & monitoring relating to intermediate indications of behaviour change

Level 4 The farmers’ opinion about extension activities

Evaluation & monitoring relating to farmers’ opinion

Level 3 Farmer participation in extension activities (participation monitoring)

Monitoring degree of participation

Level 2 Implementation of the program by extension agents (activities monitoring)

Activity monitoring

Level 1 Programming of the extension activities (inputs monitoring)

Input monitoring

Source: Modified from Bennett (1977)

In Australia, many extension programs do record inputs and outputs for accountability purposes.

But records of farmer perceptions of extension activities, changes in knowledge, attitude, skills,

motivation and group norms (KASA) and behaviour change (levels 3-6) are likely to be monitored

only informally, if at all. Sometimes observations are written in extension workers diaries and

internal files, or as verbal reports. A review of M&E of ‘Property Management Planning’ programs

in South-East Queensland showed that, while a significant amount of information was collected for

the program coordinators’ own use, few data were reported formally. In some cases the data were

86

collected for the team and reported verbally, but not documented. The most frequent time period for

reporting was six-monthly, and the type of information collected varied considerably from one

coordinator to another. It was most common to monitor levels of participation in the meetings and

farmer perception of workshop activities. Little information was routinely collected on changes

resulting from the workshops (Stewart, 1994).

Performance indicator approaches to monitoring

Traditionally, formal monitoring in program management records performance (or management)

indicators over a period of time. A performance indicator is a simple statistic recorded over time, to

inform managers of the success of some aspect of program performance. Examples of indicators in

agricultural extension might be the number of farmers contacted by extension agents per year, or the

number of members participating in discussion groups. Owen (1993) stresses the importance of

ensuring that evaluators use a full range of data collection and analysis techniques. In terms of

Bennett’s Hierarchy, a full range would include the above simple indicators, as well as some

measures of change at higher levels, such as number of farmers adopting a new practice, or decrease

in the number of farm families on social welfare.

The danger is that simplistic performance indicators can miss the bigger picture. In addition to this,

reliance on quantitative indicators alone can ‘lead’ the program: they are not neutral and can be

counter-productive and even run against the greater aims of the program. There is now a significant

debate about the past usefulness of performance indicators in government agency and corporate

planning. For example, Winston states that:

It appears that there is no evidence in the literature or in the practical experience of governments that

performance indicators have ever been successfully applied by governments to the evaluation of

human service programs, in the context of program budgeting. There is evidence to the contrary, that

exemplifies failures to get performance indicators to contribute as intended to management and

budget cycle decision making (Winston, 1991: 604).

Although many good examples of M&E exist in overseas development projects, no documented

evidence was found of comprehensive M&E in agricultural extension. On-going monitoring is

regarded to be an important part of the ‘basket of choices’ for extension evaluation. However, as

has been discussed, purely ‘indicator-driven’ approaches have serious limitations. In overseas

development projects a new approach to monitoring has been emerging that is referred to as

Participatory Monitoring and Evaluation (PM&E) (or monitoring without indicators). This approach

87

has similarities with pluralist-intuitionist approaches outlined in Section 2.4. One example of this

form of evaluation was found in Australia. Although the implementation was at an early stage at

the time of the review, it is nonetheless included.

Monitoring without indicators

Case Study 4 is an example of an innovative approach to ‘monitoring without indicators’. It seems

logical that an on-going process of data collection is necessary to allow on-going learning for a

project team. I applaud the approach referred to in this case study for managing to incorporate

qualitative data into the monitoring approach, and avoiding many of the problems of an indicator-

driven approach to monitoring. However, in comparison to many of the PM&E approaches being

practised in overseas development, the approach in this case study does not appear to provide much

opportunity for the participatory analysis of the data. It provides no guidance for how the data is

analysed. In PM&E in overseas development projects it is the stage of participatory data

interpretation that allows an opportunity for learning and dialogue concerning differences in

opinion. Despite the lack of examples available in Australia, on-going monitoring approaches are an

essential part of the ‘basket of choices’ for extension evaluation.

Case Study 4 FARMSCAPE (Farmer-Adviser-Researcher Monitoring, Simulation, Communication and Performance Evaluation) for improved management of dryland cropping

The project

FARMSCAPE is currently funded for three years and commenced in 1995. It is run by APSRU (Australian Production Systems Research Unit), a group made up of officers from the CSIRO and Queensland DPI. APSRU is directly interested in the geographical area from Northern New South Wales to Central Queensland. APSRU’s core technology is based on a computer program, Agricultural Production Simulator. APSIM uses meteorological, crop and soil data to simulate the consequences of management on system performance. FARMSCAPE’s decision support features a high degree of contextualisation and provision of insights rather than prescriptions. The notion of decision support has evolved from a sole emphasis on developing software as a ready-made product for use by farmers, to developing new relationships and learning processes withinin this project.

The FARMSCAPE project (that this evaluation concerns) was developed from a pilot project established in 1992 as part of a process to link APSRU with three farmer groups operating in the Dalby region. The aim was to demonstrate tools that were available to improve sound decision making, as well as providing feedback on those decision-making areas most likely to benefit from the products of research and modelling. The FARMSCAPE project had been operational for two years when this evaluation commenced.


The overall goal of the evaluation was to determine the degree to which various aspects of the FARMSCAPE RD&E approach was contributing to how managers think about management of their

88

production activities in the face of extra-ordinary weather uncertainty, changes in their practice, how this was happening and in what ways it could be enhanced. More specifically it aimed to:

• identify opportunities for contributing to new insights and better planning and decision-making • log evidence of the degree to which a project activity contributed to new insights and better

planning and decision-making.

Form of evaluation

This evaluation is best described as Owen’s Form 4: monitoring and evaluation for management. It is an on-going evaluation and is aimed at improving the delivery of the program as well as contributing to making summative judgements at a later stage. This evaluation process could also be described as Form 2, evaluation in program management.

Actors

Both internal and external evaluators carry out the evaluation. The audience appears to be mainly internal to the program, including the farmer clients.

Method of Evaluation

The central evaluation tool was the use of ‘logs’. In the context of this study, logs are iterative records/interviews of events, reactions, attitudes and action captured in a given framework, through a combination of:

• individual written notes/reports by key persons at selected intervals, or as an issue arises • structured debriefing sessions by selected groups themselves or facilitated by an outsider • semi-structured interviews with key persons.

Currently, 30 persons are ‘logged’ by telephone interview every three months and are interviewed face-to-face by an external evaluator every 6 months. The informants include farmers, consultants, DPI extension staff, researchers and modellers. This method of evaluation is still being developed. The intention was to trial various approaches and to select the most effective ones to use. The logs are to be used with each of the main stakeholder groups in the project.

Presentation/utilisation of findings

It is intended that the information gained in this process will be fed back to project members to assist in improving future actions.

Source: From information supplied by R. McCown, CSIRO (Australian Production Research Unit, Toowoomba) and J. Coutts (Rural Extension Centre, University of Queensland, Gatton)

Form 3: Process evaluation

The principal focus in process evaluation is on the improvement of a program and understanding

how it is being implemented. This form is generally carried out at or before program completion

and is distinct from monitoring in that it is a discrete activity. Examples of this model include

implementation studies, action research, and responsive evaluation. The Review found several

89

approaches to process evaluation, including:

• process evaluation that is carried out to understand how a program is being implemented

(illumination evaluation)

• process evaluation that is particularly responsive to the needs of the program’s clients and

aimed at improving the program while it is still developing (responsive evaluation)

• process evaluation that is carried out to provide guidance in the refinement of a program. This

approach is often carried out when reviewing pilot programs, and has a strong formative

element.

Process evaluation carried out to understand program delivery

Owen, (1993) explains how during the 1960s and 1970s the importance of examining

implementation in formal evaluation studies emerged. When programs failed to achieve desired

outcomes, practitioners started to examine how the programs had been implemented. It was found

that the type of implementation varied greatly from site to site and that the process of

implementation was an important factor in determining whether desired outcomes would be

reached. Studies of this kind are referred to as ‘process-outcome’ studies and are generally

summative in nature (Owen 1993).

Process evaluation that is particularly responsive to needs of program clients

(farmers)

Some process evaluations are designed to be responsive to the needs of those directly affected by

the program. Responsive evaluation is based on the view that those with a direct vested interest in

program delivery should control any evaluation agenda associated with the program. Often

intuitionist-pluralist approaches are conducted to in order to improve a program. Case Study 7

illustrates a process evaluation that was oriented to be responsive to the needs of the clients. This is

a small-scale formative evaluation, aimed at making the extension practice more useful for the

farmer clients. While this sort of evaluation could no doubt meet this need adequately, it might not

be appropriate for larger scale programs with many stakeholders, nor for a summative assessment of

impact. This case study illustrates the point that different sorts of evaluation are appropriate for

different purposes. However, this sort of rapid ‘formative’ evaluation based on farmer needs, is a

valuable component in the ‘basket of choices’ for extension evaluation.

90

Case Study 5 Improving the usefulness of the Pigpulse Report - an evaluation

The program

The Pigpulse Report was developed to improve the analysis of piggery data, to help decisions about interventions in piggery management. The Pigpulse project is a service provided to producers involved in the Sowtel-Pigpulse bureau operated under the project Pig Enterprise Management Services (PEMS). To generate the Pigpulse Report, a farm’s production figures and some financial records are processed using a computer package. Results of both individual farm productivity and state and district averages are sent back to farmers quarterly. The report lists how the farm production has deviated statistically from these averages. The report monitors traits such as pre-weaning mortality, average birth weights, and weaner weights.

In addition to the report, producers participate in discussion groups in a number of districts. These groups discuss issues raised by the reports, and the groups are encouraged to determine some of the topics. This evaluation concerns one aspect of the PEMS project; the Pigpulse project. This project was in the process of being developed at the time of evaluation.

Goal of evaluation

To find out if the development of the Pigpulse report was useful to producers in supporting decisions.

Form of evaluation

This evaluation best fits Owen’s Form 3, process evaluation. The evaluation information was aimed at improvement and development of an existing activity. It involved some assessment of how the activity is reaching its goals, but concentrated mainly on how the activity can be improved and made more effective. There was an opportunity to make changes in the software development as well as in the extension.

Actors

The evaluator was internal to the program, as was the primary audience for the evaluation.

Method of evaluation

The pre-test The evaluation was conducted by pre-testing an interview guide during a semi-structured interview with three producer clients. This guide contained open-ended questions and prompts that were developed from the key question and hypothesis. The interviews were conducted on the producers’ farms at a time to suit them. The interviews were arranged by telephone, the producers being informed of the interview content and purpose. The interview guide was used while the producer was asked to look at their last report and to recall how they used the report and what else was considered in making decisions in their piggery management. A copy of their Pigpulse report was provided as interviewees might otherwise say what they think they should consider instead of what they actually did consider.

During the interview, notes were written by the interviewer and were checked by the producers and modified to include their corrections. The results were coded according to a series of statements derived from the hypothesis. Examples of the statements in the coding frame were:

• whether producers read and thought through the Pigpulse report • whether trait type (of the pigs) was considered

91

• whether the magnitude of change was considered • whether targets were compared with the new level • whether the cause of change was considered • whether the effect on profit was important • whether producers changed their practice.

As a result of the pre-test, the interview guide was altered before the focus group (FGD) took place. Focus groups were chosen rather than questionnaires as the producers were feeling over-surveyed.

Use of findings

The results of the interviews were related to the level of Bennett’s Hierarchy that had been affected by the program. The report was an evaluation in process and did not include findings of the focused group discussions. However, it suggested that these results could be expressed qualitatively.

Source: Prepared from a report by Alison Spencer, Rural Extension Centre, U. Queensland, Gatton (Spencer, n.d. )

Form 4: Evaluation for design clarification

Evaluation for design clarification is generally carried out when the program is at the developmental

stage. Examples of methods used in this form are ‘evaluability assessment’ (Wholey, 1983; and

Smith, 1989), ‘program logic’ (see below). While no examples of ‘evaluability assessment’ were

found the review, two examples of program logic were found.

Program logic

Program logic is the rationale behind a program or project – i.e., what are understood to be the

cause-and-effect relationships between program activities, outputs, intermediate outcomes, and

ultimate outcomes. Usually represented in a diagram, program logic shows a series of expected

consequences, not just a sequence of events. It is best conducted as a team and is noted for enabling

staff to come to achieve consensus about the realistic outcomes and goals of a program. Ideally,

program logic would be mapped out by the project team before the implementation, and modified

and referred to throughout the life of the project. However, in many cases, program logic is

conducted later in the project life to help in the evaluation process, or to bring a program back on

track.

Program logic is a type of theory-driven approach to evaluation. It makes an attempt to open the

black-box, and understand what mechanisms underpin a program. Since this review was completed,

program logic seems to have become a much more widely used tool in extension evaluation. But at

92

the time of the review, only one such completed evaluation was found. Case Study 6 is presented

here as it is an example of the use of program logic in evaluation for the purpose of clarifying the

design of a program once it has commenced. Theory-driven evaluation approaches do appear to

have much to offer extension projects, as they help practitioners understand how programs work,

and thus can facilitate learning. When program logic is done in a team, it can help create a shared

vision of the program. Clarifying program logic can also be very useful for developing the key

evaluation questions that will guide the evaluation strategy for a large project. One example case

study was also found in which program logic was used as the starting point for an impact

evaluation.

Case Study 6: A workshop to refine the program logic for Coranagmite Salinity Program

The Program

The Victorian DNRE (Department of Natural Resources and Environment) decided to have a rigorous look at one of its programs directed against the problem of dryland salinity. This program had been running for about five years, and was considered to be at a settled stage. The specific project to be evaluated was the Corangamite salinity program.

Goal of the evaluation To gain consensus as to the project aims and to improve the implementation of the project.

Form of evaluation This evaluation best fits Owen’s Form 4, evaluation for design clarification.

Actors The group that requested the workshop was the DNRE staff responsible for this program, i.e. an internal audience. Other stakeholders included the steering committee and an implementation group, and landholders. These groups were not included at this stage. The facilitators were from the DNRE, and had recently received training in program logic.


Two workshop sessions were carried out over 1½ days. Eight people attended and the mode of inquiry was informal discussion and brainstorming. The steps taken are outlined below:

Step 1. Defining the problem: In the first session the group set about defining the problem. The facilitators encouraged participants to think critically. At this stage it became apparent that there was no common understanding of the underling salinity problem. The participants were encouraged to describe the ‘rich picture’, the political nuances, and all the stakeholders. Once they had agreed on what the problem really was, they moved on to discussing what outcome would be achieved if this problem was solved. When a consensus had been achieved on realistic outcomes, they moved on to the next stage.

Step 2. Mapping the first layer of the model. Participants were asked by the facilitator to try to forget that they were involved in the program in reality and to imagine that they were in the planning phase of a new program. They were then asked to think about what was needed in place in the hypothetical program for the outcome to be achieved. This formed the first level of the hierarchy in

93

the program logic map.

Step 3. Mapping out subsequent layers of the model, going downwards from the outcome: The participants were asked how these necessities were to be achieved, and thus the map of program logic was created, moving downwards from the desired outcome to activities necessary to achieve these. At each level of the map the facilitators would double check with the participants, by asking questions such as: “are you really sure?” and “if you do this and this, will that really lead to the outcome?” At each step of the model the cause-effect relationship was explored. The facilitators constantly questioned the participants with regards to any assumptions they had made concerning the links between cause and effect.

Step 4. Back into the real program: Participants were asked to return to the reality of the current program and to compare the conceptual model with that of current activities and state of play of the real program. It became obvious that certain things were not being done in the dryland salinity program that should have been done and also that certain inappropriate activities were on-going.

Step 5. Considering evaluation: This stage was to consider where evaluation should take place. The facilitators suggested that the links between cause and effect that involved ‘leaps of faith’ should be evaluated. Participants were asked to identify these weak links in the model. Once the locations or links had been established, the participants were asked to consider which data would answer their questions and what evidence was available to check that one link in the program led to the next intermediate outcome.

Presentation/utilisation of findings

The results of these workshops were not formally recorded. However, five days after the workshop ended, the DRNE members of the dryland salinity program were involved in an independently-organised workshop with the whole dryland salinity program team of Corangamite catchment area. The participants of the previous program logic workshop rescheduled the whole workshop to incorporate what they had learned in the previous workshop and to share it with the rest of the team. The program design was significantly altered to address the issues raised in the first workshop. This constitutes a high degree of utilisation, according to Patton’s framework (1997).

Source: pers. comm. with David Beckingsale (1997) Department Natural Resources and Environment, Melbourne.

Form 5: Evaluation for program development

Form 5, evaluation for program development, is for the synthesis of a new program. The program

will not yet have been implemented and will still be at the planning stage. The focus is on the

context of the program environment. Examples of models and methods include needs assessment,

economic evaluation, review of best practice, research synthesis, exploratory RRA (rapid rural

appraisal) or PRA (participatory rural appraisal), and the Delphi technique. While conducting the

review, I found few examples of evaluation for program development. In extension, needs

assessment is often considered as research rather than evaluation and some examples may therefore

have ‘slipped through the net’ of the study, when extension agents were asked to supply examples

of evaluation. However, Case Study 7 is an example of needs assessment using participatory rural

appraisal (PRA).

94

Rapid rural appraisal

Rapid rural appraisal (RRA) and participatory rural appraisal (PRA) are used extensively in social

and natural resource projects in developing countries. I have been involved in many PRAs and

RRAs during previous work that I conducted with development projects in India and Mexico – the

following description therefore, comes from my past experience. Both these techniques involve a

multidisciplinary team staying in the community for a period of around six days. The techniques

used in RRA include semi-structured interviews, diagramming, stories, histories, diagrams,

conceptual representations, maps, transects, seasonal calenders, time lines, flow diagrams, decision

trees, venn diagrams, pie charts and more. An important component of RRA is the reliance upon

‘triangulation’: this involves crosschecking information by talking to different stakeholders, using

different methods (interviews, diagrams, and observations) and having a multidisciplinary team

(men, women, insiders and outsiders). Triangulation is carried out to assure the validity of the

findings by strengthening/optimising the sources of information (Pretty et al., 1995). RRA and PRA

share common methodology, but the aims to which they are put differ. RRA is conducted by

researchers to capture the views of local people, whereas PRA is focuses on facilitating and

contributing to the empowerment of local people to take control over their own appraisal of

problems and opportunities, and to select appropriate action.

In the last decade, modified versions of RRA and PRA have begun to be used in industrialised

country contexts. Generally, the reliance is more upon multidisciplinary teams, and a strong use of

semi-structured interviews. Less emphasis is placed upon diagrams and pictures. In the context of

evaluation, RRA/PRA can be used to develop new programs, for process evaluation and for impact

evaluation. It is most appropriate when a high degree of client feedback is required in the

evaluation. In ‘evaluation talk’ it would be best described by naturalistic inquiry. Case Study 7 is an

example of an RRA that was carried out in order to contribute knowledge to the planning process of

new programs. Again, this evaluation is an example of how evaluation can be used for a very

specific purpose. RRA techniques appear to have good potential for drawing a range of stakeholders

into the negotiation process, and are an important tool for the ‘basket of choices’ of extension

evaluation. RRA approaches have much to offer participatory extension projects, as they can foster

ownership of the program at early stages of planning if conducted well. But RRA approaches have

their potential bias and limitations. RRA can be very expensive; as they involve resourcing a whole

team of people to work with a community. It has also been argued that the rapid nature of the

appraisals introduces bias, such as seasonal bias and spatial bias (often the people who are more

95

accessible are involved more).

Case Study 7 Understanding farmer-decision making on land use

The program

The Department of Primary Industries (DPI) and National Soil Conservation Program (NSCP) is funding a major project to hasten the exchange of knowledge and management skills amongst southern Queensland farmers on fallow management and viable farming systems. The VFSG is a DPI project team integrating extension and research personnel across southern Queensland. The project commenced in January 1991 and ran for 5 years.

To ensure that VFSG programs and products would be relevant to users, and that exchange of respective information would be efficiently conducted, farmers attitudes, knowledge and needs on farming system were researched. This was part of a research project set up because of concerns over excessive soil erosion, rainfall run-off, declining soil nitrogen fertility and soil structure, sub-optimal use of herbicides and fertilisers, and insufficient rotation of crops and pastures. The research was undertaken to ensure that consequent VFSG programs and products would be relevant to users and exchange of respective information would be efficiently conducted.


Was to:

• develop an understanding of farming systems and farmer attitudes towards farming systems in southern Queensland

• gain an overview of the important issues as seen by the people involved in the management of these relevant farming systems

• determine which issues required further investigation and who possessed knowledge that would assist in these activities

• provide a base of common understanding for further market research and action.

The concept of ‘triangulation’ was used. This involved the use of different methods to study the same problem or situation and provides a way of crosschecking data.

Form of evaluation

This evaluation best fits Owen’s Form 5, evaluation for program development. In evaluation terminology it can be described as a needs-assessment using participatory methodology.

Actors

The evaluation was carried out internally to an internal audience.


The research consisted of two phases:

• a modification of Rapid Rural Appraisal • focus groups to crosscheck issues raised in the RRA, investigate issues in greater depth and to

design a quantitative survey for future progress and impact evaluation.

The rapid rural appraisal consisted of 100 face-to-face semi-structured interviews carried out by two

96

multi-disciplinary teams. Each team consisted of four interview pairs with one member from the VFSG and one from outside. In this context, RRA was used to produce a wide overview of possible aspects, people and interactions and to obtain an impression of which issues were likely to be the important. RRA is meant to be heuristic, in that it will tell you what is there. The aims of the exercise were achieved by: reaching a wide cross-section of the relevant community, adjusting the definition of what was relevant during the exercise, and not excluding any related material from the report. The interviews were conducted in pairs (or small groups) with different backgrounds.

The focus groups were arranged into five districts and were segregated with regard to their tillage practice. Participants had to be located within 60 minutes travel time to the focus group venue and those farmers involved in the RRA were not involved. Farmers who met the above criteria were selected randomly from DPI and Grainco mailing lists. The prospective respondents were contacted by telephone and invited to attend the focused group discussions. Moderators were used in the discussions to focus the groups in a non-directional manner. Interviews were taped to assist analysis. A typist transcribed the transcripts from the focus groups and a computer package called Ethnograph was used to analyse the data from the focused group discussions and search for patterns and themes.

Use of findings

Implementation of VFSG sub-programs involving many of the recommendations from this research has commenced.

Source: Taken from an internal document by David Blacket and Gus Hamilton, DPI ,Goodndiwindi Queensland, 1992.

97

4.6 Summary of the Evaluation Samples

A summary of some main characteristics of the sample of 50 evaluation studies of extension

programs is provided in Table 3.

Table 3 Summary of main characteristics of the sample of evaluation studies

Approximate percentage in categories (n=50)

Criterion 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Owen’s form: F1- impact F2- monitoring F3- process F4- design F5- synthesis

F1

43%

F2

7%

F3

37%

F4

9%

F5

4%

Summative or formative

Summative 54% Formative 32% Both 14%

Stage of program Developing 25% Settled 58% Fin-ished 12%

Approach adopted

Largely objectives-testing approach 63% Other approaches 37%

Perspectives considered

Multi-perspective view taken 40%

Maintained a single focus-perspective 60%

Data type Qualitative 44% Quantitative 27% Both 11%

Surveys Structured survey 52%

no structured survey 48%

Rapid rural appraisal (RRA)

RRA 10%

RRA not used 90%

Focused groups Focused groups used 42% not used 56%

The case studies and the summary of all 50 samples (Table 3) illustrate that extension evaluation is

employed for several different purposes, and examples were found that cover all of Owen’s five

forms of evaluation (for program development, design clarification, process evaluation, evaluation

in management and impact evaluation). However, the most common form of evaluation was found

to be impact assessment, followed by process evaluation. Surprisingly, I found that very little

program monitoring is conducted in agricultural extension (Owen’s Form 4; evaluation in program

management). Only two examples were found during the entire review. Of the two cases of

monitoring, one was a study in the initial stages of developing a system of program ‘monitoring

without indicators’. This study had much in common with PM&E approaches seen in overseas

98

development projects. While it had potential to offer an approach to facilitate an on-going dialogue

with project stakeholders, it fell short in terms of including a participatory approach to interpret and

make sense of the data.

Several studies were found that fitted Owen’s Form 3: process evaluation. These were a mix of

small-scale formative studies to improve a program (generally by making it more responsive to

farmer needs), and summative-process studies aimed at checking to ensure the extent of program

implementation. It appeared from the time-scale of the reports that process evaluations are a

relatively recent practice.

One study was found that fitted into Owen’s Form 4: evaluation for design clarification (concerning

program logic). This approach offers insight to extension agents in understand how a program

works, and offers a process to draw a number of stakeholders into the task of clarifying real

objectives, and plausible outcomes. It is notable, that since the review was conducted in 1997, there

has been a large increase in this form of evaluation, with most evaluation training courses now

including it in their agendas. This perhaps reflects the increasing demand for extension evaluation

in general. While only one study was found that fitted Owen’s Form 5, evaluation for program

development generally goes under the name of ‘applied research’, rather than evaluation and for

this reason studies of this type may have slipped through the net of the review.

Overall only 23% of the evaluations in the study were solely formative. Indeed, the majority of the

evaluations reviewed were carried out to provide a report to justify spending and to understand

whether the stated objectives of the program had been met or not. This is reflected in that fact that

evaluations reviewed were most frequently conducted while the program was in a settled stage of

development.

In terms of methods of data collection, there appeared to be a tendency for evaluators to stick to a

handful of familiar methods and approaches in evaluating extension programs. The qualitative data

collection commonly took the form of focus groups and semi-structured interviews, while

questionnaires are by far the most frequently practised quantitative data collection tool used.

It is evident that a range of approaches to evaluation were employed in extension evaluation, with

objectives-testing approaches being most common. Most of the studies were limited to a summative

assessment of project impact, and there was a pre-dominance of objectives-testing approaches

99

(63%). However, there was a group of evaluations that went against the general trend of objectives-

testing approaches for justification of spending. These studies were much more focused on gaining

information to revise an existing project, for example Case Study 5 (process evaluation) and 6

(program logic).

Intuitionist-pluralistic approaches were to some extent, apparant in the studies that employed some

form of RRA. These studies took a pluralistic approach, and encouraged dialogue between different

stakeholders, they also included farmer participants in the interpretation of the data. While few

examples of pluralistic-intuitive were found, a multi-perspective view is apparently quite common;

this involves utilising people from different backgrounds and ‘worldview’ in conducting evaluation.

While only two theory-based approaches to evaluation were found, several studies used Bennett’s

Hierarchy of Outcomes to structure the data collection or analysis. Few examples were found that

referred to a management-decision approach, such as Patton’s (1997) ‘Utilization-focused

evaluation’.

This study did not set out to examine changes in the form or approach to evaluation. However, there

are strong indications from the sample of reports received that the range of approaches to evaluation

existing in agricultural extension is expanding. There are a number of new and innovative

evaluations under way that are generally more formative and qualitative than appears to have been

the norm in the past. Nonetheless I suggest that, while there is a range of forms and approaches to

evaluation in practice, the current suit of approaches needs to be more expanisive to meet all current

challenges and demands for extension evaluation in Australia. In particular no evaluation study was

found that facilitated on-going communication between multiple project stakeholders.

4.7 Examining the Evaluation Studies Against the Meta-

evaluation Questions

In addition to analysing the sample case studies with the help of Owen’s forms of evaluation, some

of the key meta-evaluation questions developed in Chapter 2 were applied. Other meta-evaluation

questions were not addressed due to the fact that the review was limited to an analysis of project

documentation. However, the full set of meta-evaluation questions is used to evaluate the central

case evaluation at a later stage of this thesis (Chapters 7 and 8). The questions that were addressed

100

in this review are:

• Is the evaluation relevant to the context and purpose in hand, and does it fulfil the evaluation


• Is the evaluation guided by program theory?

• Was the design and data collection/analysis of the evaluations valid?

• Are the evaluation processes socially and politically just?

Was the evaluation relevant to the context and purpose in hand?

It was not often possible to determine from the evaluation reports whether the approach employed

had been appropriate to the project context. For the most part, little attempt was made to justify the

approach taken in the evaluation studies. The reporting suggests that many practitioners were not

aware of the range of evaluation models and methods and available, and therefore would not have a

full choice in selecting models/methods to best suit the context.

Were the evaluations guided by program theory?

Chen (1990) discusses the traditional neglect of program theory and the popularity of method-

oriented evaluation. He writes:

Until very recently evaluation literature has rarely been concerned with the importance of theory in

evaluating a program or with how to incorporate theory into evaluation processes (Chen 1990: 17).

The results of this review in the field of agricultural extension concur with Chen’s findings,

although there is indication of a move away from method-driven evaluation in extension. Many of

the evaluation studies examined appeared to be heavily methods-focused, and showed little

evidence of examining the underpinning causal mechanisms of the project – or indeed questioning

the project objectives themselves – Case Study 1 provides an example of objectives-based

evaluation. In other words they were ‘black box’ evaluations, which do not comply with the

premises of good evaluation as advocated in Chapter 2.

Was the design and data collection/ analysis of the evaluations valid?

There was a wide range of data collection/analysis employed in the different evaluation studies, and

it is hard to generalise about their validity. However, anecdotal evidence from discussions with

101

various organisations and individuals during this study suggested that there was a lack of

confidence as to how to select appropriate approaches and evaluation tools, to conduct evaluations

and utilise findings. The observations suggest that in agricultural extension there is a tendency to

stick to a handful of familiar frameworks and methods such as Bennett’s Hierarchy (Bennett, 1977)

questionnaires and focus groups. In some of the studies there seems to be a lack of understanding

of evaluation in its broad sense, and as a result practitioners confine themselves to familiar methods

that may not best fulfil their objectives.

An important factor contributing to a lack of expertise and confidence in evaluation is the fact that

most extension personnel are trained primarily in the physical sciences and few have a sound

background in social science methods. Nowadays extension agents are expected, on top of the

requirements of applied science-based aspects of their work, to cope with a range of concepts from

the social sciences. Most have never studied literature on program evaluation, and if they have had

training in social science methods it has been mainly from in-service training on short courses.

There is an obvious need to address this problem if extension organisations are to improve the

ability of staff and the quality of evaluation practice.

Were the appropriate levels of the outcomes hierarchy considered?

Bennett’s Hierarchy is a hierarchy of goals established for agricultural extension programs

(Bennett, 1977) (see Table 2). This framework is generally well known to agricultural extension

agents, probably because it is mentioned in a well-used extension text (Van den Ban and Hawkins,

1988: 205-208). Bennett lists seven levels of goals in extension and claims that it becomes more

difficult to evaluate at higher levels of the hierarchy, as it becomes more difficult to show that

changes at these levels are the result of extension activity and not of other factors.

102

Level 8 Consequences for society (an additional level to Bennett’s Hierarchy) Level 7 Consequences for the target group Level 6 Behavioural changes in the target group Level 5 Change in knowledge, attitude, skills, motivation and group norms (KASA) Level 4 The farmer’s opinion about extension activities Level 3 Farmer participation in extension activities Level 2 Implementation of the program by extension agents Level 1 Programming of the extension activities

Figure 7 Distribution of evaluation studies according to the level of investigation of Bennett’s

Hierarchy

Figure 7 shows the frequency of occurrence of studies in the sample at each level of Bennett’s

Hierarchy. The majority (89%) of the evaluations considered level 3; the degree of farmer

participation in farmer activities, while 60% of the evaluations considered changes in KASA (level

5), and only 22% measured behavioural changes. Using Bennett’s Hierarchy as a starting point, any

program evaluation of extension should include some reference to:

• ‘Level five’; changes in farmer knowledge, attitudes and skills, in addition to lower level

outcomes such as farmers participation in activities. An evaluation which only addresses issues

such as participation (level 3) or reactions (level 4) would provide little evidence of worth or

merit of a program.

• ‘Level six’; changes in behaviour. Despite the fact that the higher level outcome of behavior

change is harder to attribute to the program activities alone, evaluation of these programs

0

10

20

30

40

50

60

70

80

90

%of sample

1 2 3 4 5 6 7 8

Level of Bennett's hierarchy

103

should make an attempt to provide evidence concerning the extent to which these programs

contribute toward behavioural change (which is after all the ultimate aim of extension).

The majority of evaluation studies in the sample did not examine higher level outcomes as

categorised by Bennett’s levels 5 and 6 (changes in knowledge, attitudes, skills; changes in

behaviour), but were preoccupied with assessing farmer/ client reactions, and participation rates.

Are the evaluation processes socially and politically just?

In Chapter 2, I proposed that in good evaluation, attention is paid to who is given voice in

determining the indicators for the evaluation. I proposed that participatory extension projects

involve their farmer-clients in the decisions surrounding ‘what gets measured’, and in the valuing

process itself. In the model of Fourth Generation Evaluation, it is proposed that the ‘victims’ of a

program are identified and their view sought (Guba and Lincoln, 1989). In the 50 evaluation cases,

few of the potential ‘victims’ were consulted. The majority of studies tended to be situated within

the discourse of the project; non-participants, or protagonists of the project not being consulted.

Figure 8 shows the results of scoring the evaluation studies in the sample against a continuum of

participation based on a scale of participation modified from (Dart, 1997). In 54% of the

evaluations, the clients were not heavily involved in developing the indicators for the evaluation. At

the other extreme, 7.5% of the evaluations involved the clients being facilitated to carry out their

own evaluation.

104

Participation levels adopted

1 2 3 4 5 6 7

External evaluator, no consultation

External evaluator, staff consulted

Staff conduct evaluation, clients not consulted

External evaluation, staff & clients consulted

Staff & clients develop indicators together

Clients conduct facilitated self-evaluation

Clients conduct self-evaluation - alone

Figure 8 Level of participation in evaluation by farmer clients in the studies

In many instances in agriculture, evaluators have seen little reason for client participation. However,

the proponents of highly participatory evaluation stress the relevance in many situations in

agricultural extension, of involving clients in developing the evaluation framework, especially with

regard to the choice of indicators. Patton (1997: 101) writes that such participatory evaluation ‘is

most appropriate where the goals of the program include helping participants become more self-

sufficient and personally effective’. In such instances, participatory evaluation is also intervention

oriented in that the evaluation is designed and implemented to support and enhance the program’s

desired outcomes.

Insufficient data was presented in most of the evaluation reports to allow critical comment on the

appropriate level of participation in each individual evaluation study. However, few examples (7%)

were found where farmer clients were documented as having involvement in deciding the sort of

change to be measured, despite the fact that many of the extension projects in the sample espouse

0

5

10

15

20

25

% of sample

1 2 3 4 5 6 7

Level

105

the rhetoric of participation. In this thesis I encourage those responsible for evaluating participatory

extension projects to consider more suitable or expanded forms of evaluation that involve farmers in

determining what is included in an evaluation study.

4.8 Conclusions

Program evaluation in agricultural extension in Australia (1990-1998) is used to meet a

considerable range of tasks in agricultural extension. Examples were found of all of Owen’s five

forms of evaluation. However, the majority of the studies reviewed were limited to a summative

assessment of project impact. There was also very little monitoring of projects. There was a range

of approaches to evaluation, including objectives-testing approaches, theory-based approaches and

some forms of pluralist-intuitionist approaches. But the most common approaches were objectives-

testing approaches, and these do not attempt to ‘open the black box’ and understand how a program

works.

In terms of complying with the premises of good evaluation developed in Chapter 2, I found it

difficult to fully assess the evaluations, as the review comprised an analysis of only written

evaluation documents. However, it appeared that there was a tendency towards methods-driven

approaches, a lack of knowledge of program evaluation theory, a tendency to focus of the lower

levels of Bennett’s Hierarchy, and a tendency towards black-box evaluations – which are all

contrary to the derived premises for good evaluation. Method-driven approaches imply that the

methods dictate the evaluation design, rather the opposite. Many of the studies did not appear to

consider the context and evaluation needs when deciding on the evaluation design. This was

confirmed by the tendency to stick to a few methods of evaluation, especially focus groups and

questionnaires. In terms of Bennett’s Hierarchy of outcomes, only 22% of evaluations attempted to

address changes in behaviour, which after all, is the ultimate aim of extension.

While there is a considerable range of forms and approaches to evaluation in practice, the current

suite approaches is inadequate to meet all current challenges and demands for extension evaluation

in Australia. In particular no monitoring approaches were found that could facilitate on-going

communication between multiple project stakeholders.

Evaluation as practised in agricultural extension in Australia appears to be a largely atheoretical

activity. Most of the evaluations reviewed were conducted without reference to developments in

106

‘program evaluation’ – very few examples were found that were explicit in mentioning underlying

models of evaluation and very few reports referred to the literature of program evaluation. Indeed,

in most of the sample, the evaluation design appeared to be based on concepts borrowed from

extension practice and in few cases on social science approaches. It is important that, during the

process of the review, many extension agents expressed a lack of confidence with regard to their

skills in evaluation. Nevertheless, this trend is changing, and to meet the increased demand for

extension evaluation, several training initiatives and research programs are under way that pay

attention to developments in program evaluation theory and practice from other disciplines.

In the 50 evaluation studies examined, limited reference was made to popular models of evaluation

such as Utilization Focused Evaluation (Patton, 1997), Fourth Generation Evaluation (Guba and

Lincoln, 1989), Realistic Evaluation (Pawson and Tilley, 1997) and Participatory Monitoring and

Evaluation (PM&E) (that fill the journals concerning ‘program evaluation’ and development

evaluation). These models of evaluation appear to offer much to the practice of extension

evaluation, and have potential to address demands for evaluation discussed in Chapter 3. In the light

of the findings of this review, the next step is to conduct some research on the applicability of some

these contemporary evaluation models for extension evaluation. For this reason, in Chapter 5, I

examine these popular models in more depth, and select one model to implement as a case study

(Chapter 7).

107

CHAPTER 5

CONTENDING MODELS OF PROGRAM EVALUATION FOR


Four contemporary models of program evaluation were examined in terms of the extent to

which they presented a novel approach, able to address both the macro needs of extension

projects nationally and the micro needs of the case study project across which it was to be

tested. The models examined were Patton’s (1997) ‘Utilization Focused Evaluation’, Pawson

and Tilley’s (1997) ‘Realistic Evaluation’, Guba, & Lincoln’s (1989) ‘Fourth Generation

Evaluation’, and Davies’ (1996) ‘Most Significant Change’ model. All four models were

found to have much to offer extension evaluation at a national level. Davies’ (1996) Most

Significant Change (MSC) model was selected primarily as it appeared to complement the

existing evaluation practice of the Target 10 Project and to help address the evaluation needs

of the project. The MSC model also extends current methodology, employing the collection

and selection of stories of significant change. The use of stories could offer an effective way

of creating dialogue and sensemaking between the multiple stakeholders of large extension

projects.

5.1 Introduction

In Chapter 2 I argued that approaches to evaluation should meet certain premises for good

evaluation and should be grounded in a knowledge of the evaluation needs specific to the context.

An analysis of the current context in Victorian extension (Chapter 3) revealed there is a demand for

internal evaluation that can deal with multiple audiences and promote organisational learning and

communication. An analysis of the changing nature of extension revealed the need for evaluation

that can deal with participatory, group-based extension and diverse impacts. A review of current

practice indicated a limited spread of evaluation approaches in use, with a tendency to focus on

108

methods rather than theory, and a surprising lack of project monitoring and evaluation (Chapter 4).

To address these challenges a novel evaluation model was selected, modified, implemented across a

Victorian extension project and evaluated. The aims of this chapter are to describe the selection

process for choosing one promising model of evaluation, to test as part of the empirical work of this

thesis.

I begin the chapter with an explanation of criteria that were used to select an evaluation model.

Following this, the case study project is introduced along with an analysis of the evaluation needs

and gaps for this particular project. I then provide a summary of the demands facing evaluation of

extension projects at a nation level. Next, four contending models of program evaluation are

reviewed against the evaluation needs at both the micro and macro level. Finally Davies (1996)

Most Significant Change (MSC) model, that was selected, will be outlined in more detail, and the

ways in which this model has potential to address the micro and macro evaluation needs will be

elaborated.

5.2 Criteria for Choosing a Model to Pilot

New approaches to evaluations are more likely to be useful if they are appropriate to the project-

context, and complement the project’s existing evaluation practice. The case study for this thesis

involves the implementation of a novel model of evaluation across a large extension project.

However, the aim of this thesis is not only to develop a new model for the evaluation of the case

study project, but also to develop a new model to add to the ‘basket of choices’ for extension

evaluation in Australia. Therefore, in selecting an evaluation model, the challenges identified at the

national level will also be considered.

The review of current practice in extension evaluation (Chapter 4) revealed a limited spread of

evaluation approaches in use. For this reason, I felt that it was important to select a novel approach

to evaluation that had not been widely used in extension previously. Implementing a totally new

approach can stimulate new theory and debate, especially when a limited repertoire of tools is

currently in use. In summary the criteria for selection of an approach to test included that it should:

• be a novel approach that is not currently used in extension evaluation in Australia

• address some of the macro needs of extension projects in Australia

• address the micro needs of the case study project across which it is to be implemented.

109

Macro needs of Extension Evaluation in Australia

In Chapter 3, an analysis of the changing nature of extension organisations and the implications of

these changes for the practice of extension was presented. I suggested that these changes have led to

increased demands for evaluation that can:

• show the extent to which predetermined objectives have been achieved to satisfy accountability

requirements


specific outcomes

• involve appropriate evaluation processes for participatory projects and can foster learning of the

participants and augment the intervention effort

• can be conducted as part of an internal evaluation capacity and hence foster organisational

learning

• provide performance information for communication and between multiple stakeholders

• involve processes that can foster dialogue between multiple stakeholders.

Micro needs of the Target 10 Project

The Target 10 Dairy Extension Project is one of the most established of the new genre of ‘mega’

projects that operate under the purchaser-provider model for publicly-funded extension in Victoria.

It is largely outcome focused and has a productivity goal of increasing pasture utilisation by 10%

(hence the name Target 10). It has a complex structure, and is funded by various federal, state and

industry sources as well as receiving money indirectly from levies on the sale of milk. More details

are provided concerning the background to and aims of this project in Section 7.2.

In 1998, prior to the field work for this thesis, the Target 10 Project was conducting evaluation in

the following four areas:

• evaluation of the ‘logic’ of the five core programs using Bennett’s Hierarchy and subsequent

refinement of the valued outcomes for each core program

• evaluation of each of the five core programs against valued outcomes

• economic impact analysis of the whole project

• reporting against predetermined milestones.

110

The audiences for the whole range of evaluation activities were comprised of the project staff,

purchasers, and committee members (who represented the farmer-clients and other collaborators

from the university and dairy industry).

Yet, the project management still felt that certain areas of evaluation needed to be developed.

Firstly, they wished to learn how to evaluate the overall non-economic impact of the various

initiatives and activities. This included gaining an understanding of any unanticipated or unexpected

impacts. In particular, the project funders and stakeholders alike wanted more information about the

impact that the program was having on individual farmers’ lives. They also felt that an evaluation

approach that fostered some dialogue between the various different funders and stakeholders would

be advantageous. Therefore in 1997, the major evaluation needs of the Target 10 Project were to:

• gain an overarching understanding of the impact (including unexpected outcomes) of the

project on farmers’ lives

• adopt an evaluation process that was capable of fostering dialogue and negotiation between the

different project stakeholders.

These evaluation needs correspond with some of macro needs of agricultural extension projects at

the national level (see Section 3.5). Other evaluation needs at the macro level were already being

addressed by the existing Target 10 evaluation activities. Therefore, an evaluation model was

chosen on the grounds that it met all the needs of the case study project, and some of the evaluation

needs that were associated with the changing nature of extension institutions at the national level.

5.3 Contending Models of Program Evaluation to Test

Four models of evaluation will be examined against these criteria. These models are:

• Patton’s (1997) ‘Utilization Focused Evaluation’, a decision-management approach

• Pawson and Tilley’s (1997) ‘Realistic Evaluation’, a theory-driven approach

• Guba, & Lincoln’s (1989) ‘Fourth Generation Evaluation’, a intuitionist-pluralistic approach

• Davies’ (1996) ‘Most Significant Change’ model, an intuitionist-pluralistic approach (from

development studies).

These four models were chosen on the basis of the work presented in Section 2.4 in which six

111

different approaches to evaluation were examined for what they offered to the concept of good

evaluation (experimental, objectives-testing, judgemental, decision-management, intuitionist-

pluralistic and theory-driven approaches). Decision-management, intuitionist-pluralist and theory-

driven approaches are the most contemporary, and show potential to offer new ways of evaluating

extension projects in Australia. According to the findings of the review of current practice of

extension evaluation (Chapter 4), these approaches had rarely (and only partially) been

implemented previously. Therefore, one popular model was selected from each approach, with the

exception of intuitionist-pluralist approaches, from which two models were selected. This additional

model was selected because the challenges facing development studies have similarities with those

of extension evaluation, and that there might be the opportunity for transferable learning between

the two disciplinary areas.

5.4 Utilization Focused Evaluation (UFE)

The Utilization-Focused Evaluation model was developed by Patton and is presented in his book of

the same title (Patton 1997). According to Patton:

Utilization Focused Evaluation is a system the makes evaluations useful, practical, accurate,

systematic, and ethical. This involves learning to match the evaluation approach and the design to the

information and decision needs of primary intended users taking into account other stakeholders,

political factors, organisational constraints, project/program history, available resources, and cultural

factors of a specific evaluation context. This allows for situationally responsive evaluations (Patton,

2000: 1).

The model is premised on the ability of small co-operative groups to use evaluation information to

help make decisions in order to improve programs (Rogers, 1996). As implied by the title, this

model is primarily concerned with promoting evaluation to be used by decision-makers and project

stakeholders. Patton states that:

Utilization-Focused Evaluation begins with the premise that evaluations should be judged by their

utility and actual use; therefore, evaluators should facilitate the evaluation process and design any

evaluation with careful consideration of how everything that is done, from beginning to end, will affect

use (ibid, 1997: 20 emphasis in the original).

112

In summary, Patton’s Utilization-Focused Evaluation (1997) has three particular features:

• The intended users of the evaluation are personally identified and usually brought together

(1997: 50).

• These intended users decide the focus of the evaluation, the design and the research methods,

and are involved in analysis and interpretation (1997: 376).

• All these decisions are explicitly referred to intended Utilization of the evaluation, by asking the

identified users questions such as ‘How would you use that information if we had it?’ (1997:

376).

Rogers (1996) suggests that the UFE model is fundamentally a theory of evaluation use and does

not include a prescriptive theory of knowledge or valuing. Evaluation is seen as primarily a process

of producing information that will meet an identified need for knowledge. In Patton’s words ‘unless

a lack of knowledge and information is part of the problem, program evaluation will not help an

organisation’ (Patton, 1997: 72).

Definition of evaluation in UFE

Patton defines program evaluation as:

…the systematic collection of information about the activities, characteristics, and outcomes of the

programs to make judgments about the program, improve the program effectiveness, and/or inform

decisions about future programming. Utilization focused program evaluation (as opposed to program

evaluation in general) is evaluation done for and with specific, intended primary users for specific,

intended uses (Patton 1997: 23).

Underpinning paradigm in UFE

Patton appears to subscribe largely to a pragmatic paradigm, although he is rarely explicit about

this. Instead, he refers to a ‘paradigm of choices’. Patton states that the ‘paradigm of choices

recognises that different methods are appropriate for different situations and purposes’ (1997: 279).

This avoidance of prescriptive paradigms is characteristic of a pragmatist stance. Pragmatists see

the research question as being more important than the methods used, or the world view that is

supposed to underlie the method (Tashakkori and Teddlie, 1998). For this reason, the worldview

hardly enters the picture, except in the most abstract sense.

113

Patton, like other pragmatists, rejects the forced choice between positivism and constructivism with

regard to methods, logic and epistemology. Pragmatists advocate a choice of quantitative and

qualitative methods, depending upon the research question currently posed, and the phase of the

research cycle. Pragmatists contend that science inquiry is not ‘formalistic’ and that researchers

may be both objective and subjective in epistemological orientation over the course of studying a

research question (Tashakkori and Teddlie, 1998). Pragmatists believe that values play a large role

in conducting research and in drawing conclusions from their studies. Thus pragmatists decide what

they want to research, guided by their personal value systems, and they study what they feel is

important (Tashakkori and Teddlie, 1998). This is very much in line with Patton’s arguments

A critique of UFE

To whom is the evaluator responsible?

Patton’s description of a Utilization-Focused Evaluation suggests that evaluators are primarily

responsible for meeting clients needs: ‘the purpose of a Utilization-Focused Evaluation is to answer

stakeholders’ questions’ (Patton, 1997: 43). This view contrasts with that of Scriven (1967) who

contends that the evaluator is responsible to the society as a whole, while Wadsworth (1991)

suggests that evaluators should serve the people for whom the program is designed. Patton justified

his position by stating that he is selective about the clients he chooses (by implication, he does not

have to be concerned about the intended program outcomes, since the evaluation clients are well-

intentioned) and some evaluations are done for no charge (so it is not only the well-resourced who

can be assisted with evaluation (Rogers, 1996).

Despite Patton’s reassurances, there are those that still question the ‘cosy’ relationship between the

evaluator and the client (often the project manager) and question the ability of the evaluator to

present unpalatable findings to the client in this circumstance (for example Pawson and Tilley

1997:14).

Lack of judgement of intrinsic merit of the program

Scriven (1991) disputes Patton’s view that the evaluator should act as the provider of information to

clients to help them make decisions, urging Patton to consider the intrinsic merit of the project to be

evaluated. Scriven (1986: 19) states that ‘bad is bad and good is good and it is the job of evaluators

to decide which is which’.

114

Values inquiry

Patton deals with differing values of stakeholders in an evaluation by including them in the

evaluation process. The weakness of this model is likely to be in the representativeness of the

group. Some critique UFE because of the small number of people involved in the reference group.

Action researchers, for example would criticise the model for not explicitly aiming for fuller

representation.

Applicability of UFE for extension evaluation

On the surface UFE does appear to offer a way of meeting many of the micro and macro evaluation

needs outlined. UFE also offers a novel approach in that no examples were found in the review of

extension evaluation (Chapter 4) that explicitly followed Patton’s model (yet several studies did

refer to attempting to incorporate a utilisation focus).

In terms of meeting the macro needs of extension projects in general, the UFE approach can provide

accountability and be conducted as part of an internal evaluation capacity and hence foster

organisational learning. Getting the audience of the evaluation involved in the design and

interpretation of the data ensures an opportunity for dialogue between these people. However, the

UFE model does not suggest ways of involving the beneficiaries of the program in this process,

unless they are considered an audience of the evaluation, in which case they would be represented

in a limited manner in the reference group. The methods employed would be dependent on what the

reference group decided upon, therefore this model could include data collection methods that

would help determine the impact on farmers’ lives and whatever else the group felt was important.

Therefore, UFE does meet some of the needs outlined.

Nevertheless, as Rogers points out, this model provides a theory of maximising the use of

evaluation findings, rather than a comprehensive model of how to conduct an evaluation. This is

verified by Patton’s own words:

Utilization-focus evaluation does not advocate any particular evaluation content, model, method or

theory, or even use. Rather, it is a process for helping primary intended users select the most

appropriate content, model, methods, theory and use (1997: 22).

For example, it offers no guidance as how to deal with conflicting opinions within the reference

group. It offers no suggestions for how to involve the remaining stakeholders who are not members

115

of the reference group. While it can be considered a ‘participatory approach’ insomuch as it

involves staff and funders in the evaluation design, it does not appear to stress the widescale

involvement of the farmer-clients in the evaluation process.

While UFE meets some of the needs outlined at both micro and macro level, it does not extend

evaluation in the area identified as most needed. The existing evaluation of the core programs of the

Target 10 Project already attempts to adopt a partly utilisation-focused approach, and funders and

staff have been involved in developing the evaluation plans from the beginning. For this reason

UFE was not selected to implement as part of the empirical work for this thesis.

Nevertheless, I suggest that UFE is an important contribution to the ‘basket of choices’ for

extension evaluation in Australia. Patton’s UFE provides an important reminder to all evaluators,

that no matter what evaluation model is ultimately adopted, utilisation should be considered from

the onset. It is my belief that any model of evaluation can benefit from a careful consideration of

how the findings will be used. Patton’s (1997) UFE approach has greatly influenced the overarching

concept of an evaluation bricolage. The evaluation bricolage is designed to consider utilisation, in

that it is developed on the basis on the needs of those who will use the evaluation findings.

5.5 Realistic Evaluation

Pawson and Tilley’s ‘Realistic Evaluation’ is based on a realist philosophy of science. It seeks to

evaluate programs and policies within the realities of society. It acknowledges that these programs

take place in an open system, which consists of a ‘constellation of inter-connected structures,

mechanisms and contexts’ (Kazi, forthcoming). One of the central arguments inherent in this model

is that evaluators should go beyond addressing the question: ‘does a program work’, and instead

focus on which programs work in different situations, and why. To accomplish this, Pawson and

Tilley build a model that evolves around the notion of context-mechanisms-outcome configurations

(CMOs), or configurations of mechanisms (M) operating in contexts (C) to produce outcomes (O)

(ibid 1997: 55-83).

In essence, the realist model involves three iterative steps:

116

• develop a plausible explanation for regularities between the program and the expected

outcomes by elaborating the program theory of the intervention (using CMO configurations)

• develop testable hypotheses based on these CMO configurations and test them through multi-

method approaches (i.e., compare the model to reality)

• refine the theory in terms of plausible explanations for the regularities.

The research cycle for Realistic Evaluation begins by generating a theory that is framed in terms of

propositions about how mechanisms are triggered in particular contexts to produce particular

outcomes. The realistic research cycle is presented in Figure 9.

Figure 9 The realist evaluation cycle, source: Pawson and Tilley (1997: 85)

Pawson and Tilley (1997: 216) state that ‘evaluators need to penetrate beneath the surface of

observable inputs and outputs of a program’. They suggest that observable program outcomes are

brought about by a complex range of generative forces that may not be immediately observable, and

it is the evaluator’s job to uncover these mechanisms and contexts. They focus heavily on the need

to consider context, stating that ‘evaluators need to understand the contexts within which problem

mechanisms are activated and in which program mechanisms can be successfully fired’ (ibid: 216).

Theory

Observations

Program

specification

Hypotheses

Mechanisms, Contexts, Outcomes

What might work for whom in what circumstances

Multi-method data collection and analysis on M,C,O

What works for whom in what circumstances

117

Evaluators are encouraged to seek to understand ‘for whom and in what circumstances’ a program

works through the study of the context in which the program works.

Realistic Evaluation also emphasises the need to understand the multiple outcomes of program

initiatives and how they are produced. The model suggests that programs cannot be understood as

undifferentiated wholes, but as having different effects on different subjects in different situations to

produce multiple outcomes. It is the evaluator’s job to understand how these multiple outcomes are

produced.

Definition of evaluation under Realistic Evaluation

In answer to the question of what is evaluation supposed to do, Pawson and Tilley (1997: xiv) state

that they want ‘to return evaluation to its roots in examining the effectiveness of particular social

programs targeted at specific social problems’. Thus evaluation is conducted to improve programs

and policies, to better meet stakeholder needs in order to ultimately ameliorate social problems.

Underpinning paradigm of Realistic Evaluation

Realistic Evaluation is grounded in a realist ontology. A realist ontology implies that there is a real

world that exists apart from human constructions. Realism attempts to incorporate the critique of

positivism without abandoning the concept of a knowable reality. In doing so, realism rejects the

ontological viewpoint posited by the radical constructivism approach, that suggests that there is no

objective reality. Positivism also subscribes to a realist ontology, yet there are some fundamental

differences in the notion of causality and the epistemological stance between the two paradigms.

Realism incorporates the notion of a stratified reality (i.e., a hierarchy of embedded causal orders)

with real underlying generative mechanisms (Julnes et al., 1998). Harre (1986) made a distinction

between successionist and generative theories of causation, stating that the basic task is not to

hypothesise or demonstrate the constant conjunction whereby action X produces outcome Y

(successionist causation), but rather to understand the conditions required for causal potential to be

released in practice. In contrast, positivism asserts that the world is actually equal to what we

observe, that is our experiences gained through sense impressions. Positivism incorporates a

successionist theory of causation or (Humean theory of causation) that stresses that all we can ever

observe is the constant conjunction of events, and that this is all we need to know for empirical

science to be possible.

118

Yet, realism shares with constructivism an epistemological belief that our experience of the world is

not direct, that our perception of reality is mediated by what we bring to the experience, and thus

our experience of reality is constructed (Julnes and Mark, 1998). Realism rejects the foundationalist

epistemology of positivism that sees all scientific propositions as founded on data and facts and in

which hypotheses are to be tested against the facts. Instead realism contends that knowledge is a

social and historical product, facts are theory-laden and the task of science is to invent these

theories to explain the real world and to test these theories by rational criteria developed within

particular disciplines (Manicas and Secord, 1983).

Methods in Realistic Evaluation

Pawson and Tilley (1997) talk of a need to break down the paradigm wars that have raged between

quantitative and qualitative methodologies. They advocate theory-guided evaluation, with the stress

away from a method-guided approach. They see themselves as ‘whole heartedly pluralists when it

comes to the choice of methods’ (ibid: 85). Despite all this, they give few examples of a theory-

guided evaluation that utilises qualitative approaches, although they do not rule this out.

Realistic Evaluation is presented as theory-driven approach rather than method-driven. In the case

of Realistic Evaluation, the theory relates to the CMO configurations, and the task of the evaluator

is to postulate conjectured CMO configurations drawing on folk wisdom of practitioners and formal

knowledge in literature.

A Critique of the Realistic Evaluation

In the following sections Realistic Evaluation is critiqued on the grounds:

• that it does not focus sufficiently on values, or stakeholder concerns

• that is does not focus sufficiently on utilisation of the evaluation findings

• the emphasis on testing hypothetical theories may overshadow an examination of unanticipated

program outcomes

• that insufficient guidance is given on how to choose methods, and how to generate conjectured

mechanisms

• that in some cases, a realist model may not be appropriate: (1) when the cost of conducting a

theory-driven evaluation is not justifiable; (2) when the nature of the evaluation places more

119

emphasis on answering specific stakeholder concerns.

Values

An important component of good evaluation concerns the knowledge that we construct about the

values of those affected by and interested in the programs being evaluated. I suggest that evaluation

in addition to identifying mechanisms in contexts, should include some understanding of the values

of the stakeholders. Indeed, Julnes et al. (1998: 499) argue that evaluators should consider ‘how it is

that the results of evaluative inquiry are to be translated, through the lenses of values, into decisions

about social programs and policies’. Pawson and Tilley (1997) argue that formal theories of values

do not provide a path to deciding what values should be incorporated into a particular evaluation.

This may be the case, but realistic evaluation does not offer any alternative position on how to

incorporate values into the practice of evaluation. Julnes et al. (1998: 484) suggest that ‘realist

theory must grapple with values issues and move beyond the fact-value dichotomy’.

Leviton (1994) states that theory-driven evaluation is most seriously deficient in assignment of

value to community-based programs. Realistic Evaluation does not specifically address how

stakeholders concerns will be addressed. The development of conjectured theories is directed in an

analytical manner, conducted by expert evaluators. This process of theory building does not address

the fact that, even within one sub-group of people, there may be multiple program realities (in

addition to context) and therefore multiple theories of how the program is actually working. An

alternative approach to developing a program theory during evaluation is the Action Science

approach, proposed by Argyris and Schon (1996). In this model Agyris and Schon discriminate

between practitioners’ espoused theories (what they say, or write) and theories of action (what they

do). They also highlight the fact that individual actors may hold quite distinct ‘theories of action’,

which when shared can help organisation learning. Unlike the theory advanced by Pawson and

Tilley, in Action Science these theories are not ‘collected’ but are constructed by practitioners to

understand and improve their practice.

In certain program contexts, Realistic Evaluation could perhaps be more useful if values and

alternative theories of action were addressed more fully.

Utilisation

Realistic Evaluation is not overtly concerned with issues such as how to foster ownership of an

120

evaluation study. Patton (1997) emphasised that the findings of many evaluation studied seldom get

utilised, and suggested that the stakeholders of projects must be incorporated into the evaluation

process, for the evaluation findings to be owned and ultimately acted upon. Realistic Evaluation

does not fully address Patton’s concerns (Patton, 1999).

Unanticipated/ unexpected outcomes

Realistic Evaluation does not specify how unintended or unanticipated outcomes are to be

investigated. This is common to approaches of evaluation that are based on testing key elements of

a conjectured program theory. In my experience of constructing program theory models (or program

logic) there is a tendency to get drawn into the micro mechanisms that have been postulated,

neglecting to look at the data afresh for new emerging patterns or mechanisms. Hamilton-Smith and

Hopkins (1998) suggested that in the Realistic Evaluation, no overt demand is made to include

unintended consequences once plausible and rival explanations are developed.

Clear guidance for implementation

Pawson and Tilley (1997) are vague about how to generate the conjectured theory and whose

theories should be included. There are a large number of mechanisms that could be responsible for

an outcome, and a vast number of potential outcomes. Perhaps at this stage, stakeholders could

become actively involved in deciding which theories should be tested. In three case study examples

presented in their book, researchers assembled the key hypotheses by drawing on the folk wisdom

of practitioners as well as the formal knowledge of the academy. But Pawson and Tilley do not

suggest whose views should be sought, or who should be given priority. Thus, the Realist model

does not provide clear guidance on how to generate conjectured mechanisms; it does not elaborate

clearly how evaluators should identify conjectured mechanisms nor prioritise which mechanisms

should be investigated.

In addition, Realistic Evaluation does not provide clear guidance on how methods should be

strategically selected in relation to the theory that is to be tested.

Contexts when Realistic Evaluation may not be appropriate

In some cases, depending on the nature of the evaluation questions, Realistic Evaluation may not be

appropriate. Firstly, as argued by Scriven (1994), there are situations when the cost benefit of

conducting a theory-driven evaluation may not be justifiable in terms of the benefits gained.

121

Secondly, in highly participative programs there may be instances when the nature of the evaluation

questions rightfully places more emphasis on answering specific stakeholder concerns than on

investigating carefully crafted analytical suppositions.

Summary of review of Realistic Evaluation

Despite the criticisms levelled at Realistic Evaluation, I suggest that Realistic Evaluation is a very

important contribution to program evaluation theory, offering a comprehensive and internally

consistent model for conducting evaluation. However, despite the claims that Realistic Evaluation

offers a holistic approach to evaluation, there are situations where it might be less practical and

helpful than other models of evaluation.

I suggest that the way in which the initial postulated CMO configurations are developed could be

strengthened. Pawson and Tilley suggest in an ad hoc manner that folk wisdom be consulted in

addition to literary knowledge. It is my suggestion that this step of Realistic Evaluation could be

enlarged, and directed largely by the stakeholders who are affected by program being evaluated.

Some of the work by Argyris and Schon (1996) could be incorporated at this step, and an effort

made to prioritise between the rival conjectured hypothesis (that will be tested later in the research

cycle) and that this stage consider values.

Applicability of Realistic Evaluation for extension evaluation

Realistic Evaluation is a novel approach to evaluation for extension evaluation, and no examples of

this model were seen in the review of extension evaluation in 1997 (Chapter 4). This model has

potential to meet many of the needs of agricultural extension projects in Australia. However, at the

micro level, it does not extend evaluation in the area identified as most needed.

Realistic Evaluation is able to meet many of the identified evaluation needs of agricultural

extension projects. It provides guidance for evaluation that can provide accountability for group-

based models of extension that produce diverse, context-specific outcomes. It could also potentially

be conducted as part of an internal evaluation capacity and hence foster organisational learning.

Indeed, it seems to have much to offer in terms of making sense of different outcomes in different

program contexts. Realistic Evaluation also has potential to provide performance information for

communication and between multiple stakeholders. Nevertheless, while it involves several different

sub-groups of stakeholders in testing conjectured theory, it does not provide a process for

122

facilitating this diologue, and provides no process for involving the beneficiaries or service users in

guiding the evaluation process.

At the micro level of the case study project, Realistic Evaluation does not offer evaluation processes

identified as most needed. The articulated evaluation needs of the project management were to gain

an overarching understanding of the project impact (including unexpected outcomes) and to adopt

an evaluation process capable of fostering dialogue and negotiation between the different project

stakeholders. Firstly, while Realistic Evaluation does go some way in providing information to help

stakeholders gain an overarching understanding of the impact of the project on farmers’ lives, I

have argued that it is not necessarily good at investigating unexpected outcomes. Secondly, as

mentioned above, Realistic Evaluation does not provide a process for fostering dialogue and

negotiation between project stakeholders. Realistic Evaluation is not intended to meet these

evaluation purposes. Thirdly, the existing evaluation of the Target 10 Project already included a

partly theory-driven approach – the logic of the core programs was analysed and the objectives

refined using Bennett’s hierarchy of outcomes. For these reasons the Realistic Evaluation model

was not selected for implementation across the case study project.

Nevertheless, I suggest that Realistic Evaluation would add much to the ‘basket of choices’ for

extension evaluation nationally and is a fairly comprehensive model to guide evaluation. Despite

the fact that Realistic Evaluation was not chosen, it has influenced this thesis at two levels. Firstly,

the notion of moving beyond whether a program works and focusing instead of what ‘works for

who in what situation and why’ is a central feature of the conceptual framework for good evaluation

that was developed in Chapter 2. Secondly, Realistic Evaluation was used to guide the actual meta-

evaluation (described in Chapter 8).

5.6 Fourth Generation Evaluation (FGE)

Fourth Generation Evaluation is a pluralist-intuitivist approach to evaluation presented in a book by

Guba and Lincoln of the same name (1989). Guba and Lincoln advocate a constructivist approach

to evaluation, and contend that realities are social constructs of the mind, and that there is no one

objective reality. The key emphasis of their model is on the process of negotiation, incorporating

various stakeholders more centrally into the evaluation process. Guba and Lincoln (1989) lay out

several premises for their model.

123

Firstly they state that evaluation outcomes are not descriptions of some ‘true’ state of affairs, but

instead ‘present meaningful constructions that individual actors or groups of actors form to make

sense of the situations which they find themselves in’ (1989: 8). As these constructions are shaped

by the values of the constructors themselves they are ‘inextricably linked to the particular physical,

psychological, social and cultural contexts within which they are formed and to which they refer’

(1989: 8). Therefore, colleagues who work in the same context may share the same constructions of

an event. But, reaching consensus does not imply that these constructions are any more true or valid

than other constructions.

Guba and Lincoln (1989: 9) contend that ‘evaluations can be shaped to enfranchise or

disenfranchise stakeholder groups in a variety of ways’. This includes who gets consulted in the

evaluation, who designs the evaluation, who interprets the findings and decides what is included in

the report, and to who the evaluation findings are disseminated. The evaluator should ‘interact with

humans in a manner that respects their dignity, their integrity and their privacy’ (1989: 10). In this

respect, stakeholders ‘are accorded a full measure of political parity and control’ in the evaluation

(1989: 11). In terms of commitment to utilise the evaluation findings, it is suggested that

‘evaluation must have an action orientation that defines a course to be followed’ (1989: 10). This

course of action should be negotiated between the stakeholders and should include a commitment to

follow-up the findings of the evaluation. The evaluator, therefore, plays the role of ‘orchestrator of

the negotiation process’ (1989: 10).

Definition of Evaluation under FGE

Evaluators employing Fourth Generation Evaluation judge evaluation by the extent to which it

succeeds in involving a range of participants in sharing their views. Guba and Lincoln (1989)

advocate that evaluators ought to strive for, not validity, but increasingly sophisticated constructions

of reality.

Underpinning paradigm of FGE

Fourth Generation Evaluation (FGE) employs a relativist ontology, asserting that there exist

‘multiple, socially constructed realities ungoverned by any natural laws, casual or otherwise’ (Guba

and Lincoln 1989:84). Constructivists deny that there is any reality separate from experience.

Instead, individual interpretations are thought to be all that exists. In contrast to an empiricist’s

objective stance, FGE subscribes to subjectivist epistemology. This implies that the ‘inquirer and

124

the inquired into are interlocked in such a way that the findings of an investigation are the literal

creation of the inquiry process’ (Guba and Lincoln 1998: 84).

Methodology of FGE

Guba and Lincoln (1989) advocate a ‘hermeneutic’ methodology that involves a ‘continuing

dialectic of iteration, analysis, critique, reiteration, reanalysis’, leading to a joint construction of the

case. They state a preference for qualitative methods to be employed.

A Critique of FGE

In the following sections Fourth Generation Evaluation will be critiqued for a number of reasons,

which include:

• limited documented examples of practice

• not applicable to all program contexts

• methodological dogmatism and buying into the ‘paradigm war.’

Lack of practical examples

Fishman (1992: 267) points to the lack of documented case studies demonstrating the Fourth

Generation Evaluation model. Despite all the literature, it appears that there have been few

publications documenting instances of these approaches actually being implemented. This raises

questions of how practical the FGE model really is.

Not applicable in all program contexts

To conduct a Fourth Generation Evaluation, evaluators must caution those who pay for the

evaluation that there are many realities. They will produce no recommendations, and will be

hesitant at drawing singular conclusions that could be construed as a ‘totalising narrative’. Guba

and Lincoln (1989: 45) state:

Evaluation data derived from constructivist inquiry have neither special status not legitimation; they

represent simply another construction to be taken into account in the move toward consensus.

For many clients, this sort of output from an evaluation would be unacceptable. I have seen few

tenders for the conduct an evaluation that could accommodate these stipulations. Kaplan (1994)

125

suggests that the emphasis must be on making the methodology fit the needs of the society, its

institutions and its citizens, rather than the reverse. There would be many instances when it would

not be acceptable to conduct an evaluation under the Fourth Generation Evaluation model. This is

especially true for the evaluation of large-scale extension projects that are the context for this thesis.

Indeed, Pawson and Tilley (1997: 20) suspect that Fourth Generation Evaluation ‘belongs to the

cosy suburban work of the meeting between school governors, teachers and parents in which they

thrash out homework policy for the fifth grade’. While this criticism may be unduly harsh, it serves

to point out that not all program contexts and evaluation requests could accommodate this model.

Another view on this is that models such as FGE could be usefully combined with other approaches

to meet the needs of those who request an evaluation study to be undertaken. However, Guba and

Lincoln argue against the combining of paradigms.

Methodological dogmatism and the paradigm war

The field of program evaluation, along with many of the social sciences, has become embroiled in a

‘paradigm war’. The opposing sides have variously been referred to as logical

empiricism/positivism, versus radical constructivism (Caracelli and Greene, 1997). While this war

may have created some important insights, it has also led to apparently unnecessary and unhelpful

conflicts (Julnes et al., 1998). Pawson and Tilley (1997) review the history of evaluation in terms of

this paradigm war and conclude that each of the opposing extremes was based on assumptions that

most evaluators (and others) would reject. Indeed, the battle of the paradigms seems to be fought in

academia, rather than between those who actually implement and practise evaluation. Many

evaluation textbooks encourage the would-be evaluator to decide which side they will take,

constructivism or positivism? For example, Guba and Lincoln cautions the reader:

No accommodation is possible between positivist and constructivist belief systems as they are now

formulated. We do not see any possibility for accommodation if that accommodation is to occur by

having one paradigm overwhelm the other by the sheer power of its arguments, or by having

paradigms play complementary roles, or by showing that one is simply a special case of the other

(Guba and Lincoln 1989: 17).

Despite these strong words, very few evaluation theorists advance a positivist stance, and very few

examples of radical constructivism (i.e., Fourth Generation Evaluation) have ever been

implemented (Fishman 1992). In reality most evaluation exhibits qualities of an intermediate stance.

In search of ‘peace’ in this paradigm war, some authors have suggested that a new paradigm is

needed, such as realism (Julnes et al., 1998; Pawson and Tilley, 1997) or pragmatism (Tashakkori

126

and Teddlie 1998).

Applicability of FGE for extension evaluation

FGE offers a relatively novel approach to extension evaluation that is able to address the micro

needs of the case study project, and addressing some of the needs at the macro level.

No examples of FGE were found in the review of evaluation Australian agricultural extension

(Chapter 4), although since then there have been a few studies that have certainly been influenced

by this model. For example, Roberts (1998) ‘coexistive evaluation’ of a Landcare initiative in

Queensland, and an evaluation of the Human Resource Development across the dairy industry in

Victoria (Virtual Consulting Group, 1999). However, as far as I am aware, to date the FGE model

has not been implemented in its entirety in agricultural extension. For this reason it represents a

novel approach for extension evaluation.

FGE is able to address some of the macro needs of extension evaluation. This evaluation model is

not so much focused on accountability but on negotiation and dialogue between the different

stakeholders in the evaluation. Therefore it offers processes for negotiation, but the focus is on

investigating stakeholder concerns, rather than conducting verifiable data for accountability

purposes. FGE can be conducted as part of an internal evaluation capacity (perhaps with the

addition of an external facilitator) and has the potential to foster organisational learning. FGE offers

a participatory approach to evaluation in that many stakeholders are involved in articulating their

concerns. This generally includes the beneficiaries or services users as well as specifically seeking

views of antagonists of the program.

In terms of the micro needs of the case study project, FGE is the first model that has been examined

here which appears to fully specify a process capable of fostering dialogue and negotiation between

the different project stakeholders. In terms of offering an approach that can help stakeholders

understanding of the impact, (including unexpected outcomes) if this was one of the articulated

concerns of the stakeholders then this could be investigated. In terms of capturing unexpected

negative impact, the inclusion of antagonists of the program into the evaluation, ensures that

negative views are heard. FGE offers a constructive way to involve the views of antagonists into the

evaluation, and co-opt them into the evaluation process itself.

127

In summary, the FGE model is able to meet the criteria of novelty and answering all the micro

needs of the case study project, and some of the macro needs of extension evaluation nationally. It

is suggested that FGE would contribute to the basket of choices for extension evaluation in

Australia.

However, in this instance, it is questionable whether the adoption of this approach would combine

with other approaches used in the Target 10 Project. If it was implemented wholeheartedly, and the

underpinning paradigm adopted, then it could be argued that the findings would be incommensurate

with the more objectives-based and economic models that are already practised in the case study

project. It is also questionable whether this is a practical model of evaluation, as very few instances

of this model have ever been implemented in any disciplinary field. Despite the fact that FGE meets

all the selection criteria, it was placed second to the MSC model of evaluation for applicability to

the case study project.

5.7 Most Significant Change (MSC) Model

Davies’ (1996) ‘Most Significant Change’ (MSC) model was originally named the ‘Evolutionary

Approach to Organisational Learning’. It is a qualitative approach to PM&E that involves the

collection and systematic interpretation of stories of significant change. The MSC model aims to

facilitate learning between the ‘layers’ of a large project and is based on a purposive sampling

system. Davies developed the model in an attempt to meet some of the challenges associated with

evaluating a social development project with much diversity. The structure of the MSC model was

built upon concepts of evolution theory, such as ‘survival of the fittest’, through the promotion of a

system of competitively selecting ideas (in the form of stories).

The MSC model is participatory, as many of the project stakeholders are involved in deciding the

sorts of change to be recorded. Unlike conventional approaches to monitoring, the MSC model does

not employ quantitative indicators, but is a qualitative approach. Davies suggests that quantitative

approaches to monitoring extension programs are heavily influenced by a planning ethos that places

substantial emphasis on prediction and control and that MSC is in many respects, the opposite. In

essence, there appear to be four main facets of the MSC model:

• A subjective iterative process: in which the identification of differences in interpretation is

central to the process; difference are brought to the surface and explored, not ruled out.

128

• Participatory definition of indicators: those closest to the experience being monitored (e.g.,

the field staff) are given the right to pose a range of competing interpretations of those events

without being constrained by pre-determined indicators.

• Participatory analysis of data: information is not stored or processed on a centralised basis

but is distributed throughout the organisation and processed locally. Staff not only collect

information about events, but also make their own evaluation of that information, according to

their own local perspective.

• Dynamic, rather than static indicators: Events reported reflect both a changing world and

changing sets of perceptions within the members of the organisation about what is important

within their world. For this reason, the indicators are ‘dynamic’, changing as the organisation

learns and focuses on different aspects.

The research cycle involves three main phases of action:

• establishing several ‘domains’ of change

• developing a of process based on the existing hierarchical structure of the project to collect and

review stories of change

• implementing an on-going process of collection and selection of stories.

In the first stage of the process, the evaluation audience identifies the ‘domains’ of change they

think need to be monitored at the project level. This involves a process whereby senior staff in the

project hierarchy identify broad domains, for example ‘changes in people’s lives’. According to

Davies (1996) unlike performance indicators these domains of change are not precisely defined, but

are left deliberately loose and it is initially up to field staff to interpret change belonging to any one

of these categories.

The second stage of the process involves the collection of stories of significant change (according to

the defined domains of change). The stories are collected from those most directly involved (i.e.,

farmers, extension staff and field workers). The stories are collected with the help of a simple

question in the form: ‘during the last month, in your opinion, what do you think was the most

significant change that took place in the lives of people participating in the project?’ The respondent

is then asked to give an answer in two parts. The first part is descriptive, answering the questions

‘what happened?’, ‘who was involved?’, ‘where did it happen?’, ‘when did it happen?’ In the

second part the respondent is asked ‘why do you think the change is the most significant out of all

129

the changes that took place in that month?’ and ‘what difference has it made, or will it make in the

future?’ The response to the question ‘what happened?’ generally comes in the form of a story.

These stories are reviewed on a monthly basis (see Figure 10). Each level of the program hierarchy

(i.e., field workers, regional staff, state level staff and funders) are involved in reviewing a series of

stories and selecting those that they think are the most significant accounts of change. They are then

required to document which stories they selected and what criteria they used. Davies (1996: 4)

states that ‘the monitoring system should take the form of a slow but extensive dialogue up and

down the project hierarchy each month’.

Level 3 Project funders

Level 2 State level

Level 1 Region 1 Region 2 Region 3 Region 4

Stories collected by those working directly with farmers and brought to monthly staff meetings

Represents a single story

Figure 10 Diagram to represent monthly flow of stories in MSC model for a hypothetical project

Quantification can take place at two stages. Firstly, when a ‘story’ of change is being described it is

quite possible to include quantitative information as well as qualitative information. Secondly, it is

possible to quantify the extent to which changes identified as the most significant in one location or

zone have taken place in other locations, within a specific period. However, no need is seen to

repeat this particular question every month thereafter, as in conventional monitoring systems.

At the end of each year a document is produced containing all the stories selected at the state level

over the period. The stories are accompanied by the criteria that were used to select the stories. The

document contains four chapters, each presenting three to four stories of each of the ‘domains’ of

130

change that are to be monitored by this process. However, this document is not intended to be a

final product, merely a point at which to include the funders in the process of review.

Definition of evaluation in the MSC model

Davies (1996) does not explicitly specify how the MSC model is purported to bring about better

programs. But it is clear that the MSC model was designed to enhance organisational learning.

Davies is a proponent of evolutionary theory to guide organisational learning. Campbell (1969) and

Nelson and Winter (1982) draw from the Darwinian language of evolution, adaptation, and natural

selection to explain organisational change. Under the evolutionary view of organisational learning,

whole organisations or their components adapt to changing environments by generating and

selectively adopting organisational routines.

Underpinning paradigm of the MSC model

While Davies is very much influenced by an evolutionary epistemology, he does not make the

underpinning paradigm (in terms of what can be know about reality) explicit. However, the MSC

model has much in common with constructivists’ subjective epistemology, in that the focus is on

the process of increasing stakeholders’ understanding of both the program and others’ views about

it. For example, in the MSC model, project stakeholders interpret their experiences with the project

and select instances of what they consider to be the most significant change to record as a story.

They are also required to record why this change is significant to them. Thus, when a beneficiary

tells a story of significant change, she/he interacts with the world and draws meaning from it, and it

is in the telling of the story that meaning is constructed. Then, when reviewers read and evaluate

the story, they engage with it and construct a further new meaning. When this is done in a group,

this construction may be shared. In the MSC model the criteria that are used to interpret the story

are clearly documented, made transparent and attached to the story itself. It is this transparency that

makes the whole process even more open to new and more sophisticated constructions of meaning,

because in the MSC model we know who selected each story, in what context, and for what reason.

However, Davies (1996) also subscribes to a verification stage, whereby the stories can be ‘policed’

to see if the event described in the story really occurred. This suggests that Davies does not

subscribe to a radical constructivist ontology, where ‘facts’ are considered to be a function of

multiple realities as much as values.

131

The MSC model employs inductive logic, where indicative events are abstracted out of recent

experience, and this process is renewed with each new reporting period of the monitoring system.

Instead of being deductive, it is open-ended. In terms of causation, Davies makes little attempt to

analyse the causality of the outcomes presented in the stories, or to analyse the stories en masse.

The analysis is purely based on the interpretations of the participants in the process.

Methods of the MSC model

When conventional monitoring data are analysed, they are in a form and location that strips them of

context. Typically, few text comments accompany statistics sent up from field workers. As in many

qualitative approaches, the MSC model makes use of ‘thick description’ (Geertz, 1973); closely

textured accounts of events, placed in their local context, and where the role and subjectivity of the

observer is visible. In the world of ordinary people, these often take the form of stories or

anecdotes. Within the MSC model to monitoring outlined here, these stories are accompanied by

their reviewers’ interpretations.

Within the MSC model, experience is summarised by selection rather than by inclusion; it focuses

on the exceptional rather than the common experience; and it seeks to differentiate rather than to

homogenise. Davies (1996) suggests the approach attempts to define the meaningful edges of

experience rather, than to identify central tendencies.

Critique of MSC model

The MSC model can be critiqued on the grounds that:

• in some contexts it would be inappropriate (a) where the project is small and does not have

‘layers’ (b) where the information produced by the process does not satisfy the evaluation

client’s needs

• under a realist epistemology the approach could be criticised for presenting a biased

representation of reality

• aspects of the process are overtly biased towards the view of project staff and the views of

antagonists of the project are not deliberately sought.

132

Limited contexts where the MSC model would be feasible

The MSC model aims to facilitate learning between the ‘layers’ of a large project. Clearly for

smaller scale projects, with just one hierarchical layer, this model would not be applicable in its

current form. This model is also very time consuming and requires high levels of participation that

might not be appropriate for all evaluation endeavours.

In addition the MSC model does not describe the average experience of the project participants. For

some evaluation clients, sampling the ‘outer edges of experience’ might not provide sufficient

information for accountability.

Biased representation of ‘objective’ reality

Under the lens of positivism, the MSC model can be criticised strongly for providing a poor, biased

representation of objective reality. Therefore, it cannot be seen as a valid stand-alone model to

objectively assess the overall impact of the project. However, this criticism depends very much on

the epistemological stance the critic adopts. Alasuutari (1995) makes a useful distinction between

two perspectives in qualitative research – the factist perspective (realist epistemology) and the

specimen perspective (relativist epistemology).

• Factist perspective (realist) – makes a clear cut division between the work or reality ‘out there’

and the claims made about it. A researcher with a factist perspective will need to consider how

truthful the words that the informants use are. The characteristics of language and of the

situation are only taken into account as possible noise in the channel through which information

about the world is conveyed, or as distortions in the lens through which the reality is observed

(paraphrased from Alasaturri 1995: 47).

• The specimen perspective (relativist) – the material is scrutinised in its own right. Truthfulness

and honesty are not relevant notions within which to approach the material. A specimen as a

form of research material is not treated as either a statement about or a refection of reality –

instead a specimen is seen as a part of the reality being studied. Thus honesty is an irrelevant

concept to be used in assessing the material (paraphrased from Alasaturri 1995: 63).

Under what Alasatuuri calls the ‘specimen approach’, the stories represent reality in their own right,

and it is in the telling of the stories that the storytellers and reviewers construct reality. In addition

133

to this, the interpretations that the tellers and the reviewers provide give strong clues as to how these

actors construct reality. The sharing of the stories and of the interpretations made of these stories

allows a new, shared and (hopefully) more sophisticated reality, to be jointly constructed by the

storytellers, reviewers and the story readers. In this sense then, the process helps stakeholders

apprehend reality (in its variously constructed forms) in order to make decisions.

Yet under the factist perspective, the stories themselves require crosschecking and may not

represent the best way of accurately determining what events have occurred.

Biased towards the perspective of project staff

The MSC model is situated within the existing structures of power that make a difference on a day

to day basis – not in an egalitarian setting. The sampling system privileges people participating in

the program, so it excludes the views of the antagonists of the program who might not be

participants. The interpretation of the stories is also deliberately biased towards the views of those

who interpreting the stories. Another criticism of this model (as levelled on all methods that strive

for consensus), is that particularly harsh or unpopular views may be silenced by the majority vote.

5.8 Selection of Evaluation Model to be Tested

Davies’ Most Significant Change (MSC) model for participatory monitoring and evaluation

(PM&E) was selected as the basis for developing a new model to evaluation for extension. In the

following section it will be argued that the MSC model:

• Addresses macro needs - has potential to address some of the identified evaluation needs of

Australian agricultural extension projects

• Addresses micro needs - has potential to adequately address all the evaluation needs of the case

study project

• was developed from the field of development projects which faces similar challenges common

to those of Australian extension projects

• is a highly novel in approach and methods, particularly in the collection and selection of stories

• involved the participatory interpretation of stories which is a novel way of helping stakeholders

to make sense of impact.

134

Addresses macro needs

The MSC model addresses some of the macro needs of extension projects nationally. It does not

directly attempt to verify whether valued objectives have been achieved. However, it can provide

accountability in a more ‘goal-free’ manner which could be particularly appropriate for group-based

models of extension that produce diverse, context-specific outcomes. Of all four models presented it

offers the most participatory approach which is directly orientated to foster learning by the

participants and augment the intervention effort. It was also designed to be conducted as part of an

internal evaluation capacity. The stories themselves provide performance information for

communication between multiple stakeholders.

Addresses micro needs

In terms of addressing the micro needs of the case study project, the MSC model seems very well

suited. Firstly, it offers ample opportunity for stakeholders to gain broad understanding of the

impact (including unexpected outcomes) of the project on farmers’ lives. The stories specifically

focus on the impact of the project on individual farmer lives, and the selection process is orientated

towards discussing and interpreting these stories. The MSC model also offers a process capable of

fostering dialogue and negotiation between the different project stakeholders. As in FGE, many

stakeholders from farmers to staff and funders of the project are involved in interpreting the stories

and providing feedback on these interpretations to all the other stakeholders.

Development projects face similar challenges to extension projects

International development projects include projects that concern agricultural development, as well

as health, social community development. Unlike the majority of the literature from program

evaluation per se, the evaluation literature of development studies tends to focus on internal

evaluation of externally-funded, participatory, group-oriented projects. This appears to have much

structural commonality with Australian agricultural extension projects. Many of the challenges

facing the evaluation of development projects appear similar to those facing extension projects in

Australia.

Davies developed the MSC model in an attempt to meet some of the challenges associated with

evaluating a complex micro-credit project in Bangladesh, which had both diversity in

implementation and outcomes. The project concerned was run by the Christian Commission for

Development in Bangladesh (CCDB), a Bangladeshi non-government organisation with almost 550

135

staff and an annual budget of approximately US$4 million. According to Davies (1996) the main

program CCDB runs is the Peoples’ Participatory Rural Development Program (PPRDP), that

involves more than 46000 people in 785 villages in 10 districts. Approximately 80% of the direct

beneficiaries are women. Development assistance is made available to participants in three forms:

group-based savings and credit facilities used to meet the needs of individual households, grant

assistance given to the same groups on a pro-rata basis and intended for community level

developments, and skills training, mainly for livelihood purposes. Davies (1996: 1) states that ‘the

large scale and open-ended nature of these activities poses a major problem for the design of any

system intended to monitor process and outcome’.

Thus the model was designed to be implemented across a large complex project with hierarchical

structures such as are emerging in Victorian extension under the purchaser-provider model (Section

3.4). The MSC model directly addresses the issues of how to involve many different stakeholders in

evaluation, which was one of the issues emerging as problematic for extension evaluators. The

MSC model also has built-in mechanisms for capturing unexpected change, another factor that was

stated as important by investors in extension. In addition, the participatory nature of the model is

important, as extension projects in Victoria place considerable emphasis on farmer ownership and

participation in extension projects.

Novelty

Implementing a new approach can stimulate new theory and debate, especially when a limited

repertoire of tools is in use. The MSC model represents a novel approach to test for several reasons.

Firstly, as a ‘monitoring’ approach, it represents a form of evaluation that is very rarely practised in

Australian extension (see Chapter 4). Julnes et al. (1998) suggest that monitoring is an important,

and growing form of evaluation, largely ignored by evaluation theorists. Secondly, the MSC model

is particularly novel as it is an ‘indicator-free’ form of monitoring, and challenges many of the ways

that conventional monitoring is conducted. Thirdly, the collection and selection of stories offers a

radically different approach to monitoring, that it offers much in terms of helping stakeholders

make sense of both negative and positive impact.

Stories can help stakeholders make sense of diverse outcomes

The third premise for selecting the MSC model was that it involved the collection and collective

interpretation of stories, which has been shown to be particularly effective as a ‘sensemaking’ tool.

136

Stories are used widely for co-operative inquiry and for discourse analysis, feminist research and

cultural studies. In organisational learning literature, stories are valued and studied as the preferred

sense-making currency (Weick, 1995). However, there appears to be little systematic, formal use of

stories in program evaluation. Nevertheless, program evaluation frequently involves the collection

and interpretation of stakeholder stories to make some decision or other regarding a program.

These ‘stories’ emerge during interviews (often embedded in transcripts) and in written documents

such as diaries or open-ended responses to questions. But paradoxically, there is a dearth of

literature that specifically cites the use of stories for evaluating programs.

A possibility is that the under-use of the term ‘story’ in program evaluation relates to the dubious

value in terms of revealing the truth that storytelling implies. As children we are asked by adults

whether we are telling stories – there is an implicit notion in the term ‘telling stories’, that links it to

telling fibs. In addition to truth, accuracy can also be called into question; storytelling can conjure

up the vision of the game of ‘Chinese whispers’: where a story moves around a group of people,

constantly changing and being reinterpreted as it passes from mouth to mouth. Understandably,

evaluators may question the value of collecting stories told in casual conversation, for eliciting

factual content and accurate description of events.

However, stories told in casual conversation can harness another sort of information; they provide

insight into how storytellers construct reality, and to what they attach importance. In the

organisational learning literature, stories told in casual conversation are recorded and studied by

researchers to understand organisational culture (Boje, 1991). However, they are not usually used to

drive change, or to evaluate the effectiveness of an intervention. I suggest that when the collection

of stories is coupled with a process of systematically and collectively interpreting these stories,

(including documentation of these interpretations) then storytelling can be effectively harnessed for

participatory evaluation. The interpretations themselves tell another story, and the process of

collective interpretation can have several beneficial outcomes for evaluation utilisation. Through

adding the extra step of collectively sharing and interpreting stories of program impact, a whole

new dimension to the use of stories in program evaluation is added.

Another factor contributing to the lack of use of the term ‘stories’ in program evaluation literature

probably relates to a lack of clarity with regard to what constitutes a story. In the wider literature,

stories are frequently defined by their structure. Most scholars treat narratives (or stories) as

discrete units, with clear beginnings and endings, as detachable from the surrounding discourse

137

rather than as situated events (Riessman, 1993). In Poetics, Aristotle said that stories have a

beginning, middle and end. Since that time there has been general agreement in the literature that

sequence is necessary for stories (Polanyi, 1985).

Across the literature, differences between the terms ‘story’ and ‘narrative’ are blurred and these

terms appear to be exchangeable. There is disagreement about the precise definition of a story or

narrative. According to Prince (1973), the simplest story contains three interrelated events, the first

and third state a certain situation, while the second is active. The third event depicts a change in the

state of affairs as compared to the first event. An example of this type of story is: John was a

confident driver, until he was in a crash, then he lost his confidence. However, there is a general

agreement that stories are used by the tellers to make sense of a situation. Riessman (1993) suggests

the ‘story metaphor’ emphasises that we create order and construct texts in particular contexts.

Lessons about the value of stories from the sensemaking literature

Organisational researchers emphasise the value of studying stories told in casual conversation

within organisations as part of understanding sensemaking. Louis (1980) suggests that sensemaking

is a process that uses retrospective accounts to explain surprises and make sense of change. Weick

(1995) one of the most widely cited authors in this field, suggests that the central questions for

people interested in sensemaking are how they (the agents) construct what they construct, why and

with what effects.

If storytelling is considered to be an indigenous sensemaking system in organisations, then this

medium would also seem to be suited to collectively making sense of the impact of program

interventions. Just as staff use stories told in casual conversation to make sense of surprises

(Weick, 1995), such as someone being fired in a business firm, the natural storytelling process can

be harnessed in a structured way to help practitioners make sense of the complex nuances of impact

associated with program intervention.

Storytelling is also important in terms of organisational memory. Shaw et al. (1998) contend that

stories are central to human intelligence and memory – that a good story defines relationships, a

sequence of events, cause and effect, and a priority among items – and those elements are likely to

be remembered as a complex whole. If stories about the impact of interventions can infiltrate the

collective memory of an organisation, practitioners will gain and retain a more deeply shared

understanding of what is being achieved. This could create a common base to enter into dialogue

138

about what is desirable in terms of expected and unexpected impact. Boje (1991) asserts that in

complex organisations, part of the reason for storytelling (in casual conversation) is the working out

of those differences in the interface of individual and collective memory.

5.9 Considering all Four Models

The analysis of the four contemporary models, revealed that while all the models make important

contributions, they all have certain limitations. In all cases the models would not be applicable for

every evaluation context and all contain inherent bias. This supports a central argument of this

thesis, that no single model of evaluation is sufficient for all the evaluation needs of large complex

extension projects and that combining models and methods can help off-set the bias of individual

tools. Each large project may need a ‘tailor-made’ bricolage of different methods and approaches to

meet all the needs of the people who request and use the evaluation findings. At a national level, I

argued that what is needed is a ‘basket of choices’ of evaluation models and methods from which

extension professions can pick and choose. All four models reviewed in this chapter have potential

to add to this ‘basket of choices’ and improve the quality and usefulness of program evaluation as it

is practised in Australian agricultural extension institutions.

While only one of the four models examined will be investigated empirically, it should be pointed

out that all of the models have influenced the thinking and framework of this thesis in some form or

other. Patton’s (1997) UFE approach has greatly influenced the overarching concept of an

evaluation bricolage. The evaluation bricolage is designed to consider utilisation, and was

developed on the basis on the needs of those who will use the evaluation findings. Pawson and

Tilley’s (1997) Realistic Evaluation has also influenced this thesis at two levels. Firstly, the notion

of moving beyond whether a program works and focusing instead of what ‘works for who in what

situation and why’ is a central feature of the conceptual framework for good evaluation that was

developed in Chapter 2. Secondly, Realistic Evaluation was used to guide the actual meta-

evaluation (described in Chapter 8). Guba and Lincoln’s (1989) FGE model was used to help

understand how Davies’ Most Significant Change model was intended to bring about an improved

program. At the time of the fieldwork (1997-1999), little was written on the MSC model, as Davies’

thesis was not published until 1998. As both the models are based an intuitionist-pluralist approach,

the fuller explication of FGE model contributed to my understanding how the MSC approach was

intended to work.

139

5.10 Conclusions

Four promising models of program evaluation were analysed for novelty, for their potential to

address the macro evaluation needs of extension projects in Australia, and the micro needs of the

case study project. All four models appeared to have much to offer the ‘basket of choices’ for

agricultural extension evaluation at a national level. However, Davies’ Most Significant Change

(MSC) model was selected as it seemed the best model to address the micro evaluation needs of the

Target 10 Project, across which it is to be implemented. The MSC model was also considered to be

a novel approach for extension evaluation, as the review of current practice (Chapter 4) revealed a

surprising lack of project monitoring. It was also a highly novel approach in terms of methodology,

involving the collection and participatory interpretation of stories of significant change. Indeed,

there appears to be little systematic, formal use of stories in program evaluation, yet they are

frequently studied by organisational learning specialists. I suggest that the participatory

interpretation of stories as advocated in the MSC model has much to offer stakeholders in terms of

helping them make sense of impact.

140

CHAPTER 6

METHODOLOGY FOR THE CASE EVALUATION

In this chapter the research design and methods used to evaluate Davies’ (1996) Most

Significant Change (MSC) model of monitoring and evaluation are described. The research

design involved conducting a single case study of the implementation of a modified version

of MSC model across the Target 10 Dairy Extension Project for a period of 12 months. The

MSC model was then evaluated using two different research approaches. Firstly, ‘action

research’ was conducted whereby project staff implemented, reviewed and adapted the MSC

model. Secondly, Pawson and Tilley’s (1997) ‘Realistic’ model was used to guide a meta-

evaluation and to help understand how the MSC model worked and in what situations. This

involved developing an ‘evaluation theory model’ of how the MSC process attempted to

bring about an improved project. This model was refined through an iterative process that

involved collecting and analysing empirical data. The data for the meta-evaluation included

semi-structured interviews and group discussions; observational data, project documents, and

a questionnaire.

6.1 Introduction

The empirical research for this thesis involves the modification, implementation and evaluation of

this model across an extension project. This aim of this chapter is to describe the research design

and methodology that was employed in evaluating Davies’ (1996) Most Significant Change (MSC)

evaluation model.

I begin this chapter with an explanation of the overarching research design for the case study. The

case project is then presented along with the rationale for choosing this particular project. Following

this, I describe the participatory action research component of the design. Next, a description is

provided of methodology adopted for the meta-evaluation. Lastly, the procedures that were taken to

ensure the confidentiality of the informants are outlined.

141

6.2 The Overarching Research Design

The research design consisted of a single case study of the implementation of the MSC model

across a Victorian extension project. The case study was guided by two main research approaches:

1. Participatory Action Research: whereby I facilitated the project staff to implement, evaluate and

adapt the MSC process over a period of 12 months across the Target 10 Dairy Extension

Project.

2. Meta-evaluation: whereby I evaluated the MSC model, guided by a Realistic model of meta-

evaluation. This involved developing theory about how the MSC model worked, in what

situations and why.

Key differences between the two approaches were the degree of participation of project

stakeholders and the purpose for which the information was collected. The PAR component was

conducted by the project stakeholders to meet the immediate needs of the project, while the meta-

evaluation was conducted largely by me to meet the requirements of my doctoral research. Figure

11 illustrates how the two research approaches draw data from different aspects of the

implementation process.

Another key difference between the two approaches is that the PAR component was used to provide

context-specific findings, rather than generalisable findings. The meta-evaluation was aimed at

providing both context-specific findings and developing theory concerning the worth of the MSC

model for extension projects more broadly.

142

Figure 11 A systems diagram to illustrate case-study design including concurrent action research

and meta-evaluation components

Participatory Action research component (PAR): Participants implement, evaluate and modify the MSC process in an iterative manner. This was facilitated by the researcher. The outputs of the process are used by the project stakeholders to improve the process and to make sense of project impacts. The findings of this research related to the worth of the MSC approach specifically for the Target 10 Project.

Meta-evaluation component Researcher investigates how the MSC process worked, in what situations and why. The idea was to develop theory about the worth of the MSC model for extension projects in general.

Two research approaches and purposes

Target 10 Dairy Extension Project operating in four regions of Victoria

Context

12-month implement-ation of MSC Approach

Process

Changes in knowledge and skills of participants that contribute to an improved project

Results/outcomes Outputs of Process

Information (in the form of stories and interpretations) produced by participants

Systems diagram for Implementation of the MSC process

143

6.3 Rationale for Choice of Project

The project chosen for the case study was the Target 10 Dairy Extension Project. It was selected for

several reasons that are outlined in the following section. These reasons included that the project:

• has a mega-project, with multiple-stakeholders, and was experiencing an increasing demand for

evaluation

• was one of 10 projects that had volunteered to work with the ‘Evaluation Pilot’, a group within

the Agriculture Division that aimed to facilitate projects in developing an internal evaluation

capacity

• had an established evaluation culture, making entry easier

• was staffed by people who were enthusiastic about implementing the trial

• was in a transitional stage, with plans to redevelop the project in the following year

• had a three tiered management structure.

This project has been referred to as one of the most established of the new mega-project structures

that are now being encouraged within the Agriculture Division (Kefford, B pers. comm., 1999).

Because of the challenges faced in evaluating complex ‘mega-projects’, a case project with these

characteristics of a mega-project was seen to be important. The project operates in four regions of

Victoria and has a total of 50 extension workers operating from 10 different locations. Four

‘regional committees’ and one ‘central executive committee’ oversee the project. These committees

comprise farmers, industry representatives, representatives from the universities and project staff

(elaborated more fully in Chapter 7). This constitutes a very large extension project by Australian

standards.

In the Agriculture Division of the Victorian Department of Natural Resources and Environment, a

project was initiated in 1996, entitled the ‘Evaluation Pilot’ and the Target 10 Project was working

with this Pilot project. The mandate of the Pilot Project was to facilitate over 40 mega-projects in

the Agriculture Division to develop comprehensive evaluation plans and to implement them. It was

through this project that I first gained access to extension projects in Victoria. When the fieldwork

for this thesis was being planned, the Pilot Project was currently working with only ten of these

mega-projects, all of which were part of the Agriculture Division. I was invited to informally join

the Pilot Project, and attended many of the meetings and activities of this project between 1997 and

144

2000. Several of the initial 10 projects were interested in collaborating in this research project.

From these I selected the Target 10 Dairy Extension Project as a case study.

At the time, agricultural extension in Victoria was split between two Divisions: the Agriculture

Division and the Division of Land and Water. The Division of Land and Water includes

participatory environmental initiatives such as Landcare. At the time, this division did not have an

evaluation initiative such as the ‘Evaluation Pilot’ project, therefore my access was limited to

projects within the Agriculture Division. At the time of selection of the case study project, there was

some concern that the Target 10 Project, being largely production orientated and not necessarily

‘typical’ of extension projects, might not be the most appropriate project with which to test a novel

evaluation model. However, ultimately the Target 10 Project seemed like the best choice available,

as prior to 1997, the project already had some culture of evaluation, and was probably conducting as

much if not more evaluation than any other agricultural extension project in Victoria. This allowed

a swift entry time into the project, without a need for spending time convincing management and

staff for the need of evaluation, which may have been encountered in other projects with a less

developed evaluation culture. In addition, the project managers of Target 10 Project expressed

commitment and enthusiasm for the trial.

An important aspect of the case study was the extent of change occurring at all levels. The project

was undergoing changes unrelated to the evaluation during the period studied. The project was in its

last year of the first funding round and in addition to my research an external review of ‘Human

Resources, Development and Extension, was conducted across the dairy industry. Rogers (1996)

suggests that given the rapid rate of change which is now a feature of organisations, including the

Australian public sector, it is important that an evaluation approach not be tested in a situation

where the only change occurring is that under the control of those controlling the evaluation. She

suggests that for an evaluation approach to be useful in contemporary public sector evaluation, it

must be feasible during periods of rapid change. Certainly in the case of MSC model, it was

implemented in a time of considerable change and project re-development. The fact that the project

was in re-development was also beneficial as it meant that things learned from the MSC evaluation

could more easily be assimilated into project planning.

6.4 Participatory Action Research Component

Initial reviews of published and non-published literature concerning the current practice of

145

evaluation in agricultural extension revealed that very little systematic monitoring is being

publicly reported in Australia (Chapter 4). Therefore it was not possible to conduct empirical

research into the effectiveness of an existing system of monitoring in Australia. Action research

allowed the implementation and modification of the MSC model to suit the context of the Target

10 Project in Victoria.

I felt it necessary to develop some system to ensure that the project staff had considerable

freedom in adapting the MSC model to meet the regional and project needs. Firstly, the MSC

model was designed by Davies to be implemented across a savings and credit program in

Bangladesh, and had not previously been implemented in a ‘Western’ context. I anticipated that

it would require modifications in order to adapt it to the context of a Victorian extension project.

Secondly, the Target 10 Project is also partially regionalised, with each of the four regions

having considerable autonomy with regard to project activities and organisation – so it was

important that the process could be modified to meet the regional needs. Thirdly, the MSC

model itself is designed to ‘evolve’ to the local context, with each successive round of story

review affording opportunity to refine the process itself. Consequently, I felt it was important

that the project staff themselves had freedom to evaluate and modify the process as they saw fit,

and therefore adopted a Participatory Action Research approach.

Participatory Action Research (PAR) pursues both action and research outcomes. It is most

commonly conducted with high levels of participation. Its purpose is to bring about planned

change in such a way that there is also a contribution to understanding of the system that is being

changed (Cunningham, 1993). Whyte (1991) suggests that in PAR some of the people in the

organisation under study participate actively with the researcher throughout the research process,

from the initial design to the final presentation of results and discussion of their action

implications. Whyte’s description of PAR is congruent with the research that I facilitated under

the label of PAR.

PAR is an especially participatory form of action research. Hart and Bond (1995) selected seven

criteria to distinguish action research from other methodologies, i.e., it:

• is educative

• deals with individuals as member of social groups

• is problem focused, context-specific and future-oriented

146

• involves a change intervention

• aims at improvement and involvement

• involves a cyclical process in which research, action and evaluation are interlinked

• is founded on a research relationship in which those involved are participants in the change

process.

The participatory action research that I was involved in facilitating fits with the above criteria. It

was educative in that it exposed participants to a new approach to evaluation, and to new

understandings of the impact of the project. It involved action in terms of actually implementing the

MSC model, modifying the process to meet needs, and modifying extension practice as a result of

knowledge gained through the process. The MSC process was conducted with the participation of

over 70 extension staff and collaborators, who had input into evaluating and modifying the process

as it was implemented.

Most action research is specific to the study and its findings are not necessarily generalisable. For

this reason it was conducted alongside a ‘realistic’ meta-evaluation, which is less context-specific in

orientation.

Data sources for the PAR

The MSC model was facilitated by me, but implemented across the Target 10 Project by the project

staff and committee members. Each region set in place a system for collecting and reviewing stories

of significant change on a monthly basis. The implementation process is described in Chapter 7.3.

The main forms of data that were available to the participants to modify the MSC process and to

reflect on their own extension practice are described in the following section. These were:

• stories of significant change that were collected from each region

• their own experiences of being participants in the story review process

• informal, verbal feedback detailing interpretations of the stories at the regional level meeting-

some of which was documented

• feedback documents detailing interpretations of the stories at the Statewide meetings

• a feedback document from a panel of purchasers and key influencers of the project, detailing

interpretation of their stories (but only at the end of the 12-month trial).

147

The method of developing both stories and feedback documents evolved as the process was

implemented. Therefore details of the data collection approach for each of these stages will be

discussed at greater length in Chapter 7, which includes a chronological description of the

implementation process itself.

Stories

Over the 12-month trial, 134 stories were collected and reviewed. The stories themselves were

collected in one of two ways. Some were hand-written on proformas. Others were directly narrated

into a tape recorder – this tended to happen frequently at regional committee meetings (the telling of

one story by a participant would frequently result in memory triggers for other participants). In the

latter case, mostly I transcribed the recorded stories (word for word). In a handful of cases this

process resulted in very lengthy stories, and the selection committee suggested that they be

shortened. In these cases, the stories were abbreviated and given back to the storytellers to check

and validate. The wording of the stories was never modified without the subsequent examination

and approval of the storytellers. For this reason the stories maintained a conversational tone.

However, in the final report (that was submitted to the purchasers) the project staff decided that the

stories should be edited for ease of reading, but colloquialisms and slang were not removed.

A master copy of all the stories was kept by the project management, and a spread sheet was

established on Excel to keep track of where the stories originated, who told them and whether or not

they were selected at the various review panels.

Feedback reports

At the regional meetings, stories were read aloud by the committee members, and a discussion was

held with regard to the interpretation of the stories. A summary of the discussion around the stories,

and a list of the selected stories, was included in the minutes of the meeting.

After each statewide meeting, (every three months) I compiled a feedback report. These documents

consisted of the stories that were selected plus a list of the comments that people had made about

the stories. During the meeting at which the stories were reviewed, these comments were scribed

onto a white board for all to see, and the committee members were asked to verify the comments at

the end of each meeting. Discussion at all of the state executive meetings where story interpretation

occurred, was audio-recorded using a multi-directional microphone. This was done with the consent

148

of the committee members. Where there was some confusion or lack of clarity the transcripts were

consulted, but generally the feedback documents consisted of the points made on the white board

(more detail is provided in the description of the implementation of the process in Section 7.3). The

feedback document was disseminated across the entire project – by email to all the project staff, and

to all committee members by hand or mail.

At the end of the 12 months, a meeting was held with a panel of project funders and key purchasers.

This meeting was also audio-recorded. The outcomes, and main discussion points form this meeting

were written into a formal evaluation document (Target 10 Evaluation Stories, Dart, 1999c), and

circulated to all project staff and stakeholders. This document was sent to over 250 people.

How the data were used by participants to modify the MSC model

All project staff and committee members had access to the stories and to the interpretations of the

stories from the various meetings (referred to as feedback documents). Using this information the

regional committees made decisions about how the process should operate in their regions. This

process was also facilitated by ‘regional champions’, who were responsible for encouraging the

collection of stories, and facilitating the selection process during the regional committee meetings.

In addition, three group discussions were held between all ‘regional champions’ of the process.

These meetings concerned the MSC implementation process, rather that the business of reviewing

the stories per se. This group acted as a quasi ‘action learning group’. Through discussion, this

group shared their experiences of the MSC model in the different regions to suggest modifications

to the process so that it would better meet their needs. These suggestions were then taken back to

the four regional committees for discussion, modification and in some cases were implemented.

This process is fully described in the Section 7.3.

6.5 Meta-evaluation

The Participatory Action Research (PAR) component provided context-specific findings and helped

the project staff to tailor the MSC process to meet their needs rather than generalisable theory. As

this thesis is concerned with developing a new approach to evaluation for agricultural extension

projects nationally, I felt it was also important to evaluate the implementation in a manner that

would produce some theory about the applicability of the MSC model for extension projects in

149

general. For this reason, I also conducted a less participatory ‘meta-evaluation’ of the MSC model.

Scriven (1991) describes meta-evaluation as the evaluation of evaluations; indirectly the evaluation

of evaluators. He states that meta-evaluation is the professional imperative of evaluation: it

recognisees that ‘evaluation begins at home’. Scriven states that an evaluation should be

conceptually clear, comprehensible in all detail, comprehensive, cost-effective, credible, ethical,

and explicit about the standards of merit or worth. Conducting a meta-evaluation (as in any

evaluation) is a value-laden process and will hinge upon the epistemological approach and the

criteria selected by the evaluator. Worthen et al. (1997) state that there are over 60 different models

of how evaluation should be conducted. With the same logic, there are also many models that could

inform the criteria (and the epistemology) by which an evaluation is judged (i.e., meta-evaluation).

However, Rogers (1996) states there is little guidance available for the practice of meta-evaluation:

…evaluation approaches have rarely been formally evaluated. Instead they have tended to be adopted and promoted without convincing evidence of their effectiveness or appropriateness, nor information about how they work (Rogers 1996: 8).

Because of the scarcity of guides to meta-evaluation, I developed a series of meta-evaluation

questions to guide this process (Section 2.6). All of these questions are addressed in the meta-

evaluation and presented in Chapter 9. However, in Chapter 8, the first question (did the evaluation

lead to an improved project?) was examined in detail as this was seen as a central concern.

This first question is based on the premise advocated by Rogers (1996) that good evaluation should

contribute to improved projects; that is projects which meet needs. Improved projects can come

about in one of three ways; improving existing projects, developing alternative projects, or

developing new projects. In this case, I am dealing only with the first. To address this question, I

felt it was necessary to conduct some research to help explain not only whether the MSC model

contributed to improvements in the project, but how and in what situations these ‘improvements’

occurred. Gaining an understanding of the contexts in which the model was most successful (and

where it was not successful) is also an important question to answer in order to be able to generalise

about the applicability of the model to extension projects more widely (Chapter 9).

Theory-guided approaches to evaluation involve the construction of ‘program theory models’; these

are essentially casual models that elaborate how a program is intended to meet needs. A theory-

driven approach to meta-evaluation was adopted for this research as it appears to offer great

potential in gaining a deeper understanding of how an evaluation approach is working, and whether

150

it would be applicable to other contexts. This seemed particularly appropriate for a novel approach

to evaluation such as the MSC model that has only been implemented once before in a different

culture, for a different program.

Theory-guided meta-evaluation

Shadish (1992) defines theory-driven evaluation and theory-driven meta-evaluation:

Theory-driven evaluation suggests constructing program theories to describe how programs work (Chen & Rossi 1981, 1983, 1987). These theories are usually embodied in program models. Such models draw on substantive hypothesis from various disciplines, from past research and from ordinary knowledge about program inputs, processes and outputs…. Theory-driven meta-evaluation parallels this logic, but models evaluations themselves rather than programs. Models of how evaluation operates draw from theories of evaluation, from past research, and from ordinary knowledge about the inputs, operating processes and outputs of evaluations. Theory-driven evaluation tests the plausibility of such models (ibid: 29, emphasis in original).

Several authors have suggested recently that theory-driven models of evaluation are particularly

relevant for meta-evaluation (Rogers, 1996; Shadish, 1992). In this thesis I elected to use Pawson

and Tilley’s (1997) ‘Realistic’ model of theory-driven evaluation to guide the meta-evaluation. One

of the central arguments inherent in this model is that evaluators should go beyond addressing the

question: ‘does a program work?’, and instead focus on which programs work, and in what

situations and why. To accomplish this, Pawson and Tilley build a model that evolves around the

notion of context-mechanisms-outcome configurations (CMOs), or configurations of mechanisms

(M) operating in contexts (C) to produce outcomes (O) (more fully elaborated in Section 6.5).

It should be noted that Pawson and Tilley’s realistic model was designed for evaluating programs,

not evaluations. However Shadish, (1992) suggests that logic for meta-evaluation parallels that of

evaluation, the difference being that the evaluation is modelled rather than the program. Using

Realistic Evaluation to guide the meta-evaluation appeared to be advantageous. Nevertheless, I

found no instances of a realistic model being used for meta-evaluation.

151

A realistic model was adopted for several reasons that are outlined in the following sections. These

reasons were to:

• ensure that the meta-evaluation was not a black-box venture.

• learn how the MSC model attempts to contribute to project improvements

• understand how the MSC model works and in what situations and why

• implement another promising approach to evaluation

• develop a theory about the applicability of the MSC model for extension projects in Australia.

At the time of implementing the trial, little guidance was available with regard to how the MSC

model is supposed to contribute to project improvements. This was because Davies’ thesis was not

published until 1998, which was after this trial had been conducted. The original instruction for the

process was limited to a 10-page publication (Davies 1996) that outlined the method but did not lay

out a theory of how the model was intended to bring about an improved project. Therefore, it

seemed important to develop some plausible theory as to how the MSC model attempts to bring

about change, and the Realistic model provides guidance for doing this.

An analysis of four contending models of program evaluation revealed that Pawson and Tilley’s

(1997) Realistic Evaluation had much to offer the ‘basket of choices’ for extension evaluation in

Australia (Section 5.5). Using Realistic Evaluation to guide the meta-evaluation also affords some

new knowledge with regard to how this model can be implemented.

As the aim of this thesis is to develop a model of evaluation for agricultural extension projects in

Australia, any assessment of the MSC model should not be confined to an investigation of how

successful the model was for the Target 10 case project. It should also consider how applicable it

could be for extension projects more broadly. A realistic approach to meta-evaluation facilitates

understanding of the underlying mechanisms of the MSC model and under which contexts these

mechanisms are likely or unlikely to be fired. Following this logic, a realistic approach to meta-

evaluation affords a description of how appropriate the MSC model may be for other extension

projects.

152

Method of applying Realistic approach in the meta-evaluation

Pawson and Tilley’s description of Realistic Evaluation was followed were possible. Conjectured

CMO (context, mechanism outcome) configurations for the MSC process were developed in an

iterative manner then tested against empirical evidence.

Developing an evaluation theory model and the conjectured CMO

configurations

The CMO configurations were developed after the 12-month trial, and the process of developing

them involved five main steps:

1. I developed a preliminary ‘free-form’ evaluation theory model for how I felt the MSC process

was intended to bring about project improvements.

2. 10 semi-structured interviews were conducted in which informants explained how they

constructed the MSC process. At the end of each interview, the informant and myself modified

the ‘evaluation theory’ model. At the end of 10 interviews, the final ‘free-form’ conjectured

evaluation theory model was drawn (see Figure 12).

3. I identified nine key conjectured CMO configurations from the evaluation theory model and the

interview transcripts.

4. A questionnaire was administered to all staff, with regard to the extent to which they felt that

the conjectured outcomes had been achieved.

5. A causative theory of how the model works was developed, using data from the questionnaire,

interviews, group discussions, and observational data.

I developed the first draft of the ‘evaluation theory model’ using the experience and knowledge I

had gained during the 12-month implementation of the MSC process. This was in the form of a

visual diagram ‘free-form’ diagram consisting of text boxes and arrows, similar to the final model

presented in Figure 12. I have termed this model an ‘evaluation theory model’, in reference the

equivalent ‘program theory model’. ‘Program theory models’ are developed to understand how a

program (or project) is intended to bring about intended outcomes. In theory-guided meta-

evaluation, it is the evaluation which is modelled, rather than the project, the term ‘evaluation

theory model’ is intended to reflect this difference.

153

The evaluation theory model was substantially revised over the course of conducting 10 semi-

structured interviews with key informants (interview guide is provided in Appendix 1.1). In doing

so, I tried to place my construction of how the model operated to a side - letting the informants talk

freely of how they constructed the MSC process. At the end of the interviews, I showed them my

preliminary evaluation theory model and discussed differences in the way I constructed the process

compared to how they constructed it. Together, we modified the evaluation theory model based our

mutual insights. Therefore, as the interviews proceeded the model was refined. The final

conjectured ‘evaluation theory model’ is presented in Figure 12.

The evaluation theory model depicted in Figure 12 was organised with the help of Bennett’s

Hierarchy of outcomes – a hierarchy of goals established for agricultural extension programs

(Bennett, 1977). Bennett lists seven levels of goals, which can also usefully be applied to goals of

an evaluation initiative:

• Level 7 – Consequences for the target group

• Level 6 – Behavioural changes in the target group

• Level 5 – Change in knowledge, attitude, skills, motivation and group norms

• Level 4 – The farmers’ opinion about evaluation activities

• Level 3 – Participation in evaluation

• Level 2 – Implementation of the evaluation by extension agents

• Level 1 – Programming of the evaluation activities

In Figure 12 these goals are listed down the left-hand margin. (Patton, 1997: 236) also used this

hierarchy to structure an evaluation theory model for his Utilization Focused Evaluation Model.

This evaluation theory model did not give reference to context, and was not presented in the CMO

format as suggested by Pawson and Tilley (1997). Nevertheless, during the interviews, the 10 key

informants discussed project contexts where outcomes were more likely to be realised, and why

these outcomes had occurred. Using the transcripts from these interviews, alongside the evaluation

theory model, I identified nine conjectured CMO configurations, which are presented in Chapter 8

(Table 14). In Figure 12, the numbers in the text boxes correspond to the nine key outcomes in the

CMO configurations. Some of the outcomes represented in Figure 12 were not selected to test in the

meta-evaluation. The rationale for selecting some key outcomes over others concerned the purpose

of the meta- evaluation, which was to gain some idea of the sorts of project contexts in which the

154

MSC process might bring about project improvements.

Finally, I developed a simple questionnaire to send to all stakeholders to test and refine the

conjectured CMO configurations. I developed the final model without further reference to the key

informants. The model-building process was therefore iterative and the final model was grounded in

empirical data from several sources.

In hindsight it may have been easier to develop the CMO configurations directly with the key

informants. However, both the informants and I were accustomed to drawing ‘free-form’ project

theory models, and it was natural for us to display the evaluation theory model in a visual, ‘free-

form’ manner. This enabled us to explore the connections between different conjectured outcomes.

This part of the process deviated from that laid out by Pawson and Tilley (1997). In the analysis I

also tended to focus on whether the conjectured outcomes had been achieved in different contexts,

rather than looking for evidence as to whether the mechanisms had lead to the outcomes in certain

contexts. I believe the analysis might have been richer if I had designed the questionnaire to

specifically test the CMO configurations, rather than the context to outcome relationships.

155

Figure 12 An evaluation theory model showing how the MSC process was thought to bring about improvements in the project

Purchasers reviewthe stories andreflect on desirableoutcomes

Storiescollected byfield staff andfarmers

Staff andfarmers reflecton individualpractice

Stories reviewedby committees

Committeesreflect onpractice

Discussionfrom reviewprocess isrecorded

(Feedback)

h) Short-termplanning &beneficialoperationalchanges

1. Either an improved project that can better meet needs, or improvements in the next projectUltimateoutcome

f) A moreshared visionbetween allstakeholders,making moreefficient work

h) Strategicchange andproject re-development

g) Evolution &improvementin the way theMSC processis conducted

i) Projectlearns howto continueto learnfrom pastexperience

h) More informeddecision-makingby purchasersresulting ineffective fundingallocation

Changes inbehaviourbrought aboutby the MSCprocess

e) Project learnswhat unexpectedimpacts arevalued and whichare not valued bythe differentstakeholders

c) Moraleincreasesas staffreceivefeedback

d) Management &steeringcommittee gainknowledge &change attitudesthrough theirinvolvement in theprocess

d) Deliverystaff gainknowledge &change theirattitudes frominvolvementwith theprocess

Enhancedinter-organisationalcommunicationdialogue

d) Purchasers gainnew knowledgeand attitudes aboutproject’sachievements(both expectedand unexpectedoutcomes)

KASA changes(knowledge,attitudes, skills &aspirations)brought about bythe process

Reactions tothe MSCprocess

Stories and the feedback are interesting somore stakeholder read/listen to the storiesand interpretations

a) More people pay attention to evaluativeinformation and participate in theevaluation process

Activities in theMSC Process

g) Projectlearnshow to runthe MSCprocess

The discussion is seen as valuablein terms of working out what isconsidered to be desirable impact

Stakeholders are willing to participatein the review session & commit toparticipation in the process

Farmers members also enjoythe process and feel they cancontribute to the evaluation

b) Farmers gaina voice in theevaluation & helpsteer the projectto better meetfarmer needs

As the MSC process involves existingcommittee structure farmer-committeemembers participate in the evaluation

Changes inparticipation

156

Data sources for meta-evaluation

In addition to the data collected during the trail, I also collected additional post-trial qualitative and

quantitative data. This second phase of data collected was entirely to inform the meta-evaluation; it

was not to reinforce the implementation process. The data sources are elaborated in the following

section and comprised: project documents, transcripts from story review sessions, field notes, focus

group with the purchasers, group discussions, semi-structured interviews and an internet

questionnaire.

Project documents

Relevant project documents included the output of the evaluation process that comprised 134

stories, and over 20 feedback documents, which were circulated to all staff from meetings at which

the stories were reviewed and selected. I also had access to previously-conducted evaluation reports

and the final evaluation document from the MSC model.

Transcripts from story review sessions

I systematically recorded and transcribed discussion at over 15 meetings at which stories were

interpreted by the project staff and committee members. The discussions were audio-recorded. I

later transcribed the tapes (word for word). As no discourse analysis was planned, pauses and non-

verbal sounds were not transcribed. In compliance with the ethics committee regulation all tapes

and transcripts were kept in a secure location.

157

Field notes

I kept field notes on my reflections of the difficulties and issues encountered while trying to apply

the principles and procedures of Davies’ model. After each selection committee, I recorded my

immediate reactions to the session on a tape recorder. These reactions included contextual factors

that I believed might have affected the process at that particular meeting. For example, instances

when I felt that I had done a poor job of facilitation due to my own fatigue, instances when I had an

impression that a particular person had dominated the review session. These field notes were

transcribed and entered into the computer program ‘NUD.IST’ revision 4 (Non-numerical

Unstructured Data Indexing Searching and Theorising) under the memos, and kept near the

transcripts to help make sense of any anomalies at later stage. I also noted down any insights I had

on the process as I went along, which formed part of the iterative theory building process.

Focus group with the purchasers

In August 1999 a round-table meeting was held with eight participants who were considered to be

‘purchasers’ or ‘key influencers’ of the Target 10 Project. These participants represented the Dairy

Research and Development Cooperation (DRDC), NRE, the University of Melbourne and the

DRDC Regional Development Boards. The round-table meeting took the form of a facilitated group

discussion in which all participants were asked to give their reaction to the stories. After the session

concerning the stories, the participants were taken through an ORID focus group (the guide used for

the discussion is presented in Appendix 1.2). Another person facilitated this process, to ensure that I

did not ‘lead’ the discussion.

Stanfield (1997) describes the ORID process as a technique for structuring a facilitated and focused

group conversation based four types of question:

• Objective questions

• Reflective questions

• Interpretive questions

• Decisional questions

This order of questions is designed to follow natural processes of the human mind of which we are

usually unconscious. ORIDs are used for a variety of purposes in organisations and workshops, and

is commonly used to evaluate a workshop and decide what more information is needed, to develop

158

a new vision, for strategic planning purposes or to collectively make a decision.

The reason for adopting this approach was mainly practical. Experience showed that it was difficult

to gain access to this particular group, due to their heavy time commitment to other matters. The

round-table meeting afforded a rare opportunity to speak with the seven purchasers at the same

time. The ORID method of focus group was particularly appealing in that it is good approach of

focusing conversation in a limited period of time.

Group discussions

Informal discussions were held with all the committees during the regular meeting times. The main

focus of these discussions was to identify the strengths and weaknesses of the MSC model and to

come to a decision as to whether to continue the MSC model after the 12-month trial period. In

addition a discussion was held with the regional-champions of the process.

Semi-structured interviews

Semi-structured interviews were held with eight project staff and two farmer representatives.

(interview guide is provided in Appendix 1.1). These interviews were conducted to understand how

project staff understood the MSC process to work. The data from these interviews informed the

Internet questionnaire. The interviews themselves were an important stage in developing the

conjectured CMO configurations (as described previously).

Internet questionnaire

The questionnaire (see Appendix 2) was based on conjectured CMO configurations (Table 14) that

were developed during the implementation of the MSC process and through analysis of the semi-

structured interviews. The questionnaire consisted of a mix of structured questions based on a

Likert scale and open-ended questions. In every instance the respondents were given an opportunity

to explain their ranking in comment boxes.

All project staff were sent an email alerting them to an Internet site where the on-line survey was

located. Links were placed on the survey to an ethics declaration (see Appendix 3) and to a

publication detailing the MSC model (Dart, 1999a). The surveys were anonymous, and each

informant was asked to provide a PIN number, so that any accidental duplication could be

identified. After seven and 14 days the informants were sent an email to remind them to complete

159

the survey. The survey was filled out directly onto the computer, and when the respondent had

completed the survey they pressed the ‘submit’ button, and a copy of the response was

automatically sent to me by email. The responses to the survey were copied and pasted directly into

an Excel spreadsheet.

This approach was used as I thought that it would be an interesting and efficient way of conducting

a questionnaire. All the project staff have access to a computer and Internet, and use email as a

regular part of their work.

Data analysis

Much of the data analysis was conducted in an iterative manner; as one piece of data was collected

and interpreted, this informed the design of the subsequent phase of data collection. For example,

the knowledge of the implementation of the MSC model led to the semi-structured interview guide

and preliminary theory, and the findings of this led to the formulation of questions for the

questionnaire. However, all the qualitative data was finally entered and organised using the

computer program ‘NUD.IST’ revision 4. I read and re-read the data as it was collected. I then

coded text extracts under the emergent and predetermined categories. The predetermined categories

were based on the key meta-evaluation questions developed in Chapter 2, and as the model was

constructed, the conjectured outcomes of the MSC model were also added into the coding frame.

The quantitative data collected from the on-line questionnaire was organised into spread-sheets

using the computer program Excel and statistical analysis was conducted using the computer

program SPSS. Analysis was conducted using cross-tabulations with the Gamma measure of

correlation. In order to test the level of significance of the associations, the Gamma inferential

statistic was used. Gamma is a PRE (proportional reduction of error) measure. De Vaus (1995: 168)

suggests the Gamma measure is the most common statistic when both variables in a crosstabulation

are nominal and is an appropriate test to use when there are few categories as was the case with the

data.

160

6.6 Ethical Considerations

The study was approved by the University Ethics Committee, and endorsed by all five steering

committees of the Target 10 Dairy Extension Project. In all cases, where face-to-face interviews,

group discussions or telephone interviews were conducted, the informants were presented with an

information sheet that detailed their rights to confidentiality (Appendix 4). After providing this

information, the informants were asked to give their consent for the interview to be used in this

study, this verbal consent was recorded on tape along with the interview or group discussions.

Achieving written (signed) consent in this research was seen as an unnecessary hindrance to the

researcher obtaining freely given information in a relaxed, informal environment. All the

interviewees have been involved with this process for over a period of one year, and were very

familiar with the process and the aims of the assessment of this tool. It seemed ‘overly formal’ to

obtain signed consent for reviewing a process in which they had been so extensively involved in

implementing.

Interviews were conducted in private rooms, with no other people present. The interviews were

recorded with an audio tape recorder. The tapes were transcribed word for word by me. The

transcripts and tape copies have been kept in a locked drawer in the Institute of Land and Food

Resources. The files on the computer are protected by a password. In accordance with what was

said to the interviewees, quotations are used in the thesis, but the authors of the quotation will not

be individually referred to. The quotations were sourced in terms of the job of the informant (ie.

quote: extension staff). In order to maintain the confidentiality of the few managers and regional

coordinators all were referred to as ‘project co-ordinators’.

161

CHAPTER 7

CASE EVALUATION OF THE MSC MODEL AS

IMPLEMENTED ACROSS THE TARGET 10 PROJECT

In this chapter the case evaluation is presented, comprising a description of the Target 10

Project and the implementation of a 12-month trial of the Most Significant Change model.

The output of each step of the process is described and examples are provided to illustrate

the points made. The output included 134 stories and feedback from the review of the stories

during 15 project meetings. The output also included an analysis of the 134 stories, which

formed part of the process itself. The problems and issues that arose while implementing the

model are discussed and include issues concerned with competition, time, culture and ethics.

7.1 Introduction

The previous chapters established the need for new approaches for evaluating extension projects in

Victoria, and that forms of PM&E practised in overseas development may offer promising

approaches for adoption in Australia. Davies’ (1996) Most Significant Change model (MSC), in

particular, appears to have potential to meet these challenges. In this chapter the 12-month

implementation of the MSC model across the Target 10 Dairy Extension Project is described. This

case evaluation was conducted for two reasons: firstly to investigate the potential of the MSC model

to bring about extension projects that can better meet needs; secondly, to determine what

modifications are needed to adapt this approach to suit the context of Australian extension projects.

The chapter begins with a description of the Target 10 Dairy Extension Project and the historical

background to the project. I then explain the process used in introducing the MSC model to the

project and facilitating its implementation. I draw attention to the ways in which the application

deviated from that of Davies, and the reasons for these modifications. The output of each step of the

162

process is described and examples are provided to illustrate the points made. Lastly, I discuss some

of the problems that occurred during implementation concerning competition, time, issues of culture

and ethics. In Chapters 8 and 9, an evaluation of the MSC model is described. This meta-evaluation

attempts to determine the effectiveness of MSC model and its ability to address the challenges

associated with the evaluation of the new genre of agricultural extension projects in Australia.

7.2 Description of the Project

The Target 10 Project is one of the most established of the new genre of ‘mega’ projects that

operate under the purchaser-provider model for publicly-funded extension in Victoria. It is largely

outcome focused and has a productivity goal of increasing pasture utilisation by 10% (hence the

name Target 10). It has a complex structure, and is funded by various federal, state and industry

sources, as well as receiving money indirectly from levies on the sale of milk. It is seen by some as

an example of how the mega-project structure can succeed. For example, the director of the

Agriculture Division in NRE, Victoria pointed out that the ‘success of Target 10 was the precursor

to the move to mega-projects’ (Kefford, B. pers. comm., 1999). Other industries are now adopting

similar models.

Aim of Target 10 Project

The Target 10 Dairy Extension Project was initiated in 1992, with the aim of enhancing the viability

of the dairy industry through programs that profitably increase consumption of pasture by cows. It

operates across four regions of the State of Victoria in Australia. In 1996 the project focus was

broadened to include other areas that were of high priority to the industry. Information about these

areas (grazing management, business, dairy cow nutrition, soils and fertilisers and natural resource

management) is extended to farmers through courses, discussion groups, newsletters, comparative

analysis, field days, focus farms and demonstrations and other media. Target 10 is a collaborative

industry project that aims to deliver a customer-focused change program in a coordinated and

timely manner and with community ownership. The project has a documented conceptual

framework for the delivery of the project components (see Figure 13).

163

Figure 13 The service delivery model of the Target 10 Dairy Extension Project (Boomsma et. al.,

1996)

Figure 13 depicts individuals involved in the project participating in a learning process, surrounded

by a peer group and a community supportive of change. The model recognises that, in the adoption

process, the creation of the environment for change is as important as providing the technical

information upon which the change is based (Boomsma et al., 1996). The framework involves

interaction at three levels:

• At a community level, the project provides opportunities for stakeholders to participate in the

development, organisation, and delivery of project elements, as well as in communication

strategies and collaborative alliances at state and regional levels.

• At a farmer peer group level, the project provides opportunities for creating awareness of the

issues being addressed, together with structured opportunities for peer group support and for

new learning.

164

• At an individual farmer level, the project provides opportunities to adapt management practices

to suit individual circumstances by integrating management changes into the whole farm

business and providing ongoing reinforcement for practice change.

The model represents a move away from the notion of extension being simply a technology transfer

activity, to one of industry development involving the management of both the technical

information and the participatory environment of the project. This movement is characteristic of

many newer extension projects in general and is discussed in Section 2.3. An internal Target 10

report stated that:

Combining educational components (courses) with on-farm adoption (discussion groups and visits),

follow-up (newsletters and evaluation) provides a potentially greater impact than any of these

activities in isolation or partial combination (DRDC, 1994).

Historical development of the Target 10 Project

Target 10 was originally conceived as a regional project, focusing on the benefits to dairy farmers

of maximising the utilisation of pasture grown on their farms. In 1990 Agriculture Victoria

prepared a submission to the DRDC to establish a statewide extension project based on grazing

management. In 1991 a meeting of industry representatives supported the concept and established

an executive committee to guide the implementation of the project. This meeting comprised

representatives of Agriculture Victoria, milk manufacturers, and processors, Victoria College of

Agriculture and Horticulture, Herd Improvement Organisation of Victoria, United Dairy Farmers of

Victoria and farm management consultants. In 1992 agreement on resourcing was reached between

the key financial backers, Agriculture Victoria and DRDC, and the project was initiated (Boomsma

et al., 1996).

It was anticipated that increases in production and productivity worth $50 million a year after five

years would flow to farmers and hence to the state economy and all sectors of the dairy industry as

a result of the project (Boomsma et al., 1996). In the 1980s, farm gate prices for manufacturing

milk had been static in real terms and the manufacturing sector had entered a period of declining

export prices. Domestic support for the manufacturing milk process was expected to decrease

further as the sector continued to experience increased exposure to fluctuations in the international

market place. Industry agreed that the best opportunity to increase farm profitability was through

165

improved consumption of pasture. O’Brien and Hepworth (1993) showed that pasture consumption

could be substantially improved through improved management practices.

The project that evolved proposed to extend results of pasture research and farmer experience in

grazing management. The aim was to achieve a substantial improvement in pasture consumption

and it was estimated that a 10% increase in pasture consumption on a typical farm could increase

gross returns in the order of $16,000 (Boomsma et al, 1996).

Project architecture and complexity

The organisational structure under which the Target 10 Project operates is complex. The project

has a ‘mega-project’ structure, considerable collaboration and co-delivery of programs, and is

operating under the new environment of the purchaser-provider model. It is also a large project

with over 50 staff and an annual budget of over $1,000,000. Several purchasers and co-providers

interact in a dynamic project environment (McDonald and Kefford, 1998).

Farmer ownership

From the onset, emphasis was placed on ownership of the project by the dairy farmers that it was

designed to benefit. An internal Target 10 report states that:

Target 10 was developed as a regionalised project, bringing the development, management, and

implementation and evaluation processes closer to farmers through regional committees (Boomsma

et al, 1996).

From the beginning it was envisaged that evaluation procedures would be regionalised and involve

farmers as much as possible.

Marsh and Panel (1997) point out that that a uniquely Australian situation exists in the Research and

Development Corporations: whereby Australian farmers contribute through industry-levied funding

to agricultural research, development and extension, and so have expectations of a degree of

control. The co-funding of the Target 10 Project by the DRDC with the Agriculture Division of

NRE was designed to encourage farmer participation.

166

Outcome focus

Critics of the Target 10 extension project suggest that it is overly focused on production at the

expense of community, environmental, and social issues (e.g. Beilin, 1998). An external review of

the project in 1998 recommended that it become more ‘learner-focused’ (The Virtual Consulting

Group, 1999). As described in Section 3.3, the Target 10 Project combines seemingly contradictory

aspects of both ‘outcome-focus’ and ‘learner-focused extension’. As well as having a mandate to

increase production, it attempts to be highly participatory in planning, delivery, and evaluation.

The contract to implement the MSC process within the Target 10 Project

Between May 1998 and May 1999 I introduced the Target 10 Dairy Extension staff to the MSC

process, which was then implemented, as far as I know, for the first time in Australia. The MSC

model became referred to as the ‘Story Approach’ by project stakeholders. Through implementation

of this approach the project hoped to fulfil two purposes:

• to supplement other evaluation approaches by collecting qualitative data about the impact of the

project as a whole

• to promote organisational learning within the project team and between the myriad of

stakeholders.

The audience of the evaluation comprised project staff, purchasers and committee members (who

represented farmer clients and other collaborators from the university and dairy industry).

Since its inception in 1992, Target 10 had been experimenting with evaluation. Around 1992 an

evaluation consultant introduced the project team to Bennett’s Hierarchy of (Bennett, 1975). A

specialist in evaluation was appointed to the project team who had a background in marketing and

as a result considerable work was done on developing evaluation plans at the program level.

However, in 1997 the project management still felt that certain areas of evaluation needed to be

developed. Firstly, they wished to learn how to evaluate the overall non-economic impact of the

various initiatives and activities, ie. was Target 10 more than the sum of its parts? They sought to

understand the unexpected impacts. Messages and questions of help were placed on Evaltalk1 and

1 EVALTALK’ is an international e-mail discussion group that is affiliated with the American Evaluation

Association and has over 1,500 subscribers.

167

considerable research was done. Many evaluation experts seemed to come up with different

answers, but non-of these answers seemed appropriate (McDonald, B. pers. comm., 1996).

Secondly, the project funders and stakeholders alike wanted more information about the impact that

the program was having on individual farmers’ lives. In 1992 extensive benefit-cost analysis had

been completed and individual programs had been evaluated against their objectives. However, in

1998 the project manager informed me that there was still a feeling that some of the project impact

and outcomes were not being captured. He explained that when the economic analysis came back

with a figure of $62 million net economic benefit to the dairy industry (Appleyard, 1996a) a typical

stakeholder response was ‘well that’s great, but I want to know who has got that money, and what

effect this is having!’

Around about the same time, I was searching for an extension project with which to carry out a case

study. I was interested in the Target 10 Project, as it was a well-established mega-project that

already had culture of evaluation. After discussing the evaluation needs of the project with the

management I proposed to trial Davies’ approach to participatory monitoring and evaluation

(PM&E). The Project Manager gave provisional agreement on the condition that all four regional

committee meetings and the state executive committee endorse the approach. I attended all these

committee meetings over the next three months, presented the approach, and gained the

endorsement from all the committees. In May 1998 an agreement was made that the MSC model

would be implemented across the whole Target 10 Project for a period of 12 months.

The MSC process was to be an internal, formative process of monitoring and evaluation. It formed

one part on an overall evaluation strategy that included other forms of evaluation. The Target 10

Project is conducting evaluation in the following five areas:

• internally conducted evaluation of the ‘logic’ of the five core programs using Bennett’s

Hierarchy and subsequent refinement of the valued outcomes for each core program

• internally conducted evaluation of each of the five core programs against valued outcomes

• externally conducted economic impact analysis of the whole project

• internally conducted reporting against predetermined milestones

• internally conducted evaluation of the overall non-economic impact of the project.

This MSC process addressed only the evaluation of overall non-economic impact of the project.

168

While the idea of implementing the MSC process was mandated by the project management, the

project funders had previously stipulated that this sort of evaluation should occur. Internal

documents from the Agriculture Division’s purchasing group suggest that formative evaluation

occur mainly under the proviso of continuous improvement and assessment of unanticipated

outcomes (NRE, 1997).

7.3 Implementation of the Approach

Chapter 5 provides a description of the MSC model according to Davies (1996). While parts of the

MSC model remained unchanged, the implementation in this case evaluation did involve some

modification of the process to fit the context of the Target 10 Project. However, Davies (1996)

main three steps remained central in the process:

• establish domains of change

• set in place a process to collect and review stories of change within the organisational hierarchy

• hold an annual round table meeting with the project funders.

In the Target 10 case, three additional steps were added. Firstly, as staff were unsure of the sort of

stories required, an additional process referred to as the ‘taste test’ was developed prior to

establishing the domains of change. Secondly, for the approach to evolve to meet the regional

requirements a reference group was established consisting of myself and the ‘regional champions’.

Thirdly, the Target 10 management also requested a secondary analysis of the stories en masse,

which appears to be absent from the Davies case in Bangladesh. Thus a final step was added which

involved a secondary analysis of the stories. The resulting steps were:

1. ‘Taste test’ and become familiar with the approach.

2. Establish four domains of change.

3. Establish a learning set, that includes ‘champions’ from each region.

4. Set in place a process to collect and review stories of change within the organisational

hierarchy.

5. Hold an annual round table meeting for the project funders to review the stories.

6. Conduct a secondary analysis of the stories en masse.

In the following sections, each of the above steps, as implemented in case evaluation, are described.

169

I describe how the process differed from the Bangladesh case, including the output at each stage

and provide some examples of the output to illustrate the main points.

Step 1 - ‘Taste testing’ the process

After the Target 10 regional committees had endorsed the trial, I had the task of catalysing the

implementation of the process. Davies (1996) provides no concrete guidance on how the approach

should be initiated nor gives detail as to how the first stories were collected. He explains that MSC

model was built upon previous systems of monitoring, and developed in consultation with many of

the project staff. In the Target 10 Project there was no history of monitoring or participatory

evaluation, although considerable evaluation work had been completed. Part of Davies lack of

prescription can be explained by the ethos of the approach. He suggests that the MSC process

should evolve organically to meet the needs of the organisation, and thus deliberately avoids

providing a normative framework (Davies, R., J. pers. comm., 1999).

During early attempts to initiate the process, it became apparent that staff and committee members

were not sure what sort of stories were required and asked for guidance on the length of the story,

the subject matter and the form it should take. In line with the evolutionary ethos of the approach, I

felt that the project stakeholders themselves should determine these things. Staff were

understandably nervous about putting pen to paper with so little guidance, and my initial attempts to

collect stories failed. Staff explained to me that they had no idea what was expected, and thus were

reluctant to attempt relating a story. On the recommendation of several staff, I developed a

proforma (see Figure 14) to help collect the stories, and then held a pre-trial workshop in which we

‘taste tested’ the approach with the whole project team, which consisted of over 50 extension staff

and managers.

To develop the proforma I asked three staff to help me generate a sample of stories. They agreed

that I could tape this conversation. In line with the process that Davies described (1996), I asked

the staff the following question:

‘During the last month, in your opinion, what do you think was the most significant change that

took place as a result of project activities?’

After they had described the event, I asked the following two questions:

170

Why do you think the change was the most significant?

What difference has it made, or will it make in the future?

I then transcribed their responses from the audio recording, and thus generated the first three stories.

On the basis of these stories, I developed a provisional proforma along the lines of the above

questions, but including some questions concerning things such as who was involved, where did it

occur (see Figure 14). Davies suggests that enough information should be collected so that the

actors in the story could be located and that the events could be verified. This transparency proved

important in terms of credibility of the stories, but posed ethical issues (discussed in Section 7.4).

The process of developing a proforma was iterative and was conducted in conjunction with the

reference group (see below).

Story title: ‘…………………………………………...……………. ‘ Domain:

ο changes in decision-making skills ο changes in on-farm practice ο changes in profitability/ productivity ο other significant changes

Name of person recording story: ……………………………………………………. Region: ……………………………………………………. Date of narration: … /… /…… Where did this happen? ……………………………………………………. When did it happen? ……………………………………………………. ********************************************************************************** What happened?

Why do you think this is a significant change?

What difference did it make already/ will it make in the future?

Figure 14 Items contained in the proforma for collection of stories (normally more space is

allocated for responses)

In May 1998, I held a workshop at which all the project staff were present. During this workshop, I

asked everyone to submit one story about the most significant change that they knew about, that

occurred as a result of project activity. They were all given the blank proforma. About 30 stories

were collected (hand written). These stories were placed on a wall and every staff member was

171

asked to select three stories that they considered to represent the most significant change. They did

this by placing sticky labels next to the favoured stories.

I then read aloud the highest scoring stories to the whole group. This was followed by a facilitated

discussion concerning why they had selected certain stories over the others. This discussion was

audio-recorded, summarised, and sent to all staff as a memory jogger. This process was used to get

the ball rolling. In this taste test I did not include the domains, but left it entirely up to the staff what

sort of stories they wanted to record. However, during the discussion, staff explained that it was

very difficult to select one story over another as they were about such entirely different things. At

this stage I re-introduced the concept of domains of change, which the participants recognised as an

important addition to the process.

Step 2 - Establishing the domains of change

‘Domains of change’ are loose categories that are used to distinguish different types of stories. For

example, one of the four domains used in Bangladesh was ‘Changes in People’s Participation’.

Davies suggests that, unlike performance indicators, the domains of change are not precisely

defined but are left deliberately fuzzy; and it was initially up to field staff to interpret what they felt

was a change belonging to any one of these categories. Davies (1998: 279) explains that the term

‘domain’ was borrowed from Spradly’s (1979) The Ethnographic Interview. Davies borrowed the

concept of fuzzy from the mathematical term ‘fuzzy sets’.

In Bangladesh, the domains were established by a group of project managers. In the case of the

Target 10 Project, the domains were established using the Delphi technique (see Section 6.4), and

involving over 100 stakeholders of the project. Delphi is a form of interactive (postal) surveying

that utilises an iterative questionnaire and feedback and provides participants with an opportunity to

revise earlier views based on the response of other participants, until some desired level of

consensus is reached (Cary and Salmon, 1976). Part of my rationale for using the Delphi approach

was concerned with balancing the need to have ownership of evaluation by those participating in

the process.

When evaluation questions are not established in a participatory manner, insufficient attention may

be paid to addressing projects’ felt-information needs (Leviton, 1994). During wide consultation

with the project staff, it was put to me that developing the domains of change in an analytical

172

manner, without widespread consultation could have led to a lack of ownership and the feeling that

the evaluation was being done to them, rather than being done by them. In the Target 10 case it was

extremely important for the project stakeholders to have the say about what sort of changes should

get measured. Feedback from the process indicated that stakeholders were keen to be involved in

establishing the domains.

An open-ended questionnaire was sent to 150 stakeholders asking what sort of things they felt were

important to monitor (see Appendix 5). The results of the preliminary survey are presented in

Appendix 6. In the second iteration of the questionnaire, participants were asked to rank the

domains of change in terms of relative importance (Appendix 7). Table 4 illustrates the results of

the second round of the Delphi survey. In the first round of the process, 104 replies were received

(70% response rate) and in the second round 76 replies were received (50% response rate).

Table 4 Results of the Delphi process used to establish domains of change

Domain of Change Number of Votes

Changes in on-farm practice 60

Changes in profitability 57

Changes in productivity 24

Changes in decision-making skills 16

Changes in participants’ skills 15

Changes in participants’ ability to deal with changing industry conditions 15

Changes in quality of life 14

Changes in goals and aspirations 8

Changes in on-farm environmental conditions 6

Changes in farmer confidence 5

Changes to the industry in general 2

Changes in attitudes towards Target 10 2

Because several respondents pointed out that productivity is a part of profitability and none of the

respondents selected both ‘productivity’ and ‘profitability’, it was decided to select ‘changes in

profitability’ and ‘changes in decision-making skills’ (the fourth highest ranking) rather than both

‘profitability’ and ‘productivity’. During the last round, several people pointed out that all 12

domains of change offered were important to monitor. For this reason, the fourth domain of change

has been left ‘open’ to allow stories that concern issues or events that are not covered by the first

three specific domains. Davies (1996) also used an open domain in Bangladesh.

173

The domains of change chosen for monitoring are listed below, followed by discussion of the

reasons for choosing these domains:

1. Changes in on-farm practice (received a total of 60 votes).

2. Changes in profitability (57 votes).

3. Changes in decision-making skills (16 votes).

4. Any other significant types of change.

As we used the selected domains, we came to realise that their logical order was ‘changes in

decision-making skills’, ‘changes in on-farm practice’, and ‘changes in profitability’. Thus, the

domains were usually presented in this logical order rather than in the order given above. At every

Central Executive Committee meeting over the 12 months, each region was be encouraged to

submit four stories, one for each domain of change.

Step 3 - Establishing a reference group

A reference group was established to capture learning, to encourage the adaptation of the process to

local conditions and to co-ordinate the process. While Davies describes a project member who

acted as his counterpart and states that each region developed their own process; he does not refer to

having such a reference group. After consultation with the project management it was agreed to

appoint regional monitoring ‘champions’. During the first three months of the trial, I facilitated the

collection and selection of the stories at the regional level, after which, these regional monitoring

champions took over the process. However, I continued to facilitate the selection process at all five

of the state executive meetings during the 12-month trial.

The monitoring champions volunteered for the role of co-ordinating the story collection and

selection in their region and formed the basis of a reference group for the implementation of the

process across the project. The group met on three occasions during the 12-month trial and I

communicated with them frequently on an individual basis. Modifications to the process were

discussed and decided upon during communication with these individuals. In some cases, we

decided to test an idea in one region, before recommending the practice to the other regions. A

formal learning set appeared to be an important addition to the MSC process, and is congruent with

the ‘Action Learning’ approach that characterises Davies’ work with evolutionary theory. In the

174

following sections when I use we I am referring to this group of people.

However, there were difficulties in getting the group to meet regularly due to their disparate

geographic locations. In addition there was flux in the membership of the group as people changed

roles within the project. I am of the opinion that this group could have played a much stronger and

important role in the process if it had been more stable and accessible.

Step 4 - Collecting and reviewing the stories of change

Collection of stories

In June 1997, all staff and committee members were supplied with blank proformas and were

encouraged to generate stories of significant change. We intended that these initial stories would be

shared at the staff meetings at each of the four regions. However, as very few stories were

generated using this method, we encouraged staff to share the stories verbally during the meetings.

These impromptu stories were later recorded and transcribed. For some people, this was their

preferred form of storytelling; thus each regional champions elected to purchase a tape recorder and

to record stories at staff meetings. However, some participants continued to write the stories onto

the blank proformas (see Figure 14) and the mode of initial recording was left up to personal choice.

Use of the domains

Initially we asked the storytellers to classify their story according to the four domains of change.

The domains were placed on the top of the proforma, the idea being that the storyteller would tick

the appropriate box (see Figure 14). We had envisaged that the domains would guide the story

collection process. In practice this did not occur; and the domains were not seen to be helpful in

terms of collecting the stories. However, we found that when the stories were ‘shared’ at the staff

or committee meetings, the group could categorise the stories by domains. This enabled the voting

process to run much more smoothly. Nominating domains also encouraged the story reviewers to

consider the story in some depth, and this appeared to be a useful process.

Structure of the review process

We decided at an early stage of the implementation that the MSC process should ride on the back of

the pre-existing project structure. This was considered important, as stakeholders did not want to

schedule any additional meetings. My initial proposal for the story selection design was a copy of

175

the Bangladesh structure, within the limitations posed by the pre-existing Target 10 hierarchy. I

proposed that selection would occur at four hierarchical levels (staff meetings, regional committee

meetings, state executive meetings, and purchaser meetings) and that stories were to be reported on

a monthly basis. In the Bangladesh case, the approach was implemented at four hierarchical levels,

on a monthly basis.

However, fairly early in the implementation process modifications were made to suit the local

context, particularly issues of logistics, participation, and timing. My initial proposal was for the

stories to be collected primarily by staff, based on their own experience, or second hand from

farmers and other stakeholders. The storytellers were to nominate the appropriate domain for their

story. At staff meetings, participants were to review all the stories collected over the month and to

select four, one for each domain, that represented the most significant change from their

perspective. The four selected stories were then to be sent to the corresponding regional committee

meeting. As these committee meetings were held every three months, the idea was that 12 stories

(four from each of the three monthly meetings) would be sent to the respective regional committee

meeting. Each of the four project regional committees, was then to select four stories (one from

each domain) to send to the central executive committee (CEC) meeting. They in turn would select

a further four stories at each executive meeting, that would be sent to an annual round table meeting

with the purchasers of the project. This proposal is illustrated in Figure 15.

In practice we found that the first level of selection; monthly staff meetings, was problematic in

that:

• not all regions had monthly staff meetings, and the committee structure varied considerably

• project staff were keen to include stories directly from the committee members

• collecting stories at every monthly staff meeting was too frequent

• some felt that the stories should be selected by the regional committee members, rather than at

‘staff only’ meetings.

As a result, in most cases the selection process began at the regional committee, and thus occurred

at three, rather than four hierarchical levels. Each region developed its own system of selecting and

collecting stories, depending on the existing committee structure. The only condition was that the

three large regions should submit four stories to each state-executive meeting and that they should

document how they had selected these stories.

176

.

a) Four domains of change are agreed upon using a postal process

b) People working in the field collect stories that they consider to be the most significantaccounts of the agreed domain of change

c) Stories are reviewed on a monthly basis at regional staff meetings. Four stories, one for each ofthe four domains of change, are selected and sent to the respective regional committee meeting

d) The regional committee reviews the stories selected in the region since the last meeting(usually three months, so there would be 3 x 4 =12 stories). Then four stories, one for each of

the four domains of change are selected and sent to the Central Executive Committee

e) At the Central Executive Committee meeting, four stories from each region are presented.The Central Executive Committee will select one story for each domain of change

g) Key influencers and purchasers read the document and scorethe stories in terms of the extent to which the stories represent thesort of outcomes that they wish to purchase

f) At the end of the year, a document is written containing all the stories thathave been selected by the Central Statewide Executive Committee. Thedocument will also contain the reasons for selection of these particular stories

At each stage of selection, the reason that the particular stories were selected must be recorded

Figure 15 Proposed main steps for implementation of MSC process with Target 10 Project

177

In the North East, which is a particularly small region, committee meetings are held every six

months. Only two staff work in this region, so the story selection and collection process was only

conducted at the regional committee level on two occasions over the 12-month period. However,

the two staff did submit at least two stories to every State Executive Committee, even though these

stories had not always been reviewed by all of the committee.

In the Northern Irrigation Region, the process stayed fairly faithful to the original proposal. The

only change was that stories were shared, but not selected, at the staff meetings. In the first two

staff meetings, staff tried to vote on what were effectively each other’s stories. Some felt that this

competitive element was not helpful. This point is further discussed in Section 5. After discussion

with the reference group, the system was changed so that only the regional committee level actually

reviewed and selected stories. Regional committee members were also encouraged to submit

stories directly to the regional committee. After successfully piloting this modification we agreed

that the process would be adopted in all regions.

In the region of Gippsland, the project structure is quite distinct from other regions in that it has

three sub-regional committees. These sub-regional committees became the main collection points

for the stories. The stories were not shared at staff meetings, but were reviewed and selected solely

at committee meetings dominated by farmers.

In the South West, the committee structure is again different. They favour large (over 40 members)

regional committee meetings which all the staff attend, in addition to other farmers and

representatives. These meetings are held every two to three months. As all the staff are present, it

would have involved considerable duplication to share the stories at the staff meeting. Thus the

stories were collected and reviewed at the regional committee level, and no activity occurred at the

staff meeting.

Selection process

The process by which the stories were actually selected is not detailed in Davies’ publications. In

this case we developed a facilitated process. Firstly, if the stories had not all been allocated

domains, which often happened at the regional level, each story was read out aloud and

immediately allocated a domain. The titles of the stories were written on a white board under the

178

respective domains. When all the stories had been read out, all the stories within one domain would

be considered together. The facilitator would then ask a series of questions to prompt discussion

(see Appendix 8), before moving on to a vote by hands. Each committee member was given one

vote for each domain. When the vote was done, if there was no consensus, then further discussion

was facilitated until an agreement had been reached as to which story should be selected.

Occasionally no agreement could be reached, therefore either two stories were selected, or no story

was found to be suitable. The idea was to come to an agreement as a group. As well as selecting a

story, the committee members were also asked to state why the story had been selected above the

others. Much of the discussion revolved around explanations of why they thought one story was

particularly valuable or particularly misleading (see Text box 1 for an example of discussion).

Similar discussion was held about each story prior to selections being made. This discussion was

recorded on tape, or by a note taker. The intention was that these notes would be promptly

summarised and circulated to all stakeholders in the region.

179

(Background: The story concerned a farmer who had started using a project recommended practice but then reverted to his original practice. The discussion was held at a regional committee meeting at which five farmers, one dairy representative, three staff members and the facilitator were present.)

G: It’s a story in reverse, an actual thing that happened. It’s a really important story and we need to think about the implications of this.

Facilitator: What are the implications of what happened in this story? G: But….. It is not a feel good story, that’s what I am saying C: The whole point of evaluation is that we should hear stories like this – both positive things and

negative things. M: Yes, I think this is a really valid story, I think that we need more stories like this. Facilitator: But I need you to think about what are the implications of a story like this? F For the farmer or us? Facilitator: For both S: Target 10 pushed him too far.. G: No no no no no, he’s got to be confident to carry it out. [all talk at once] C: He believes it enough in the first place, but he had a lack of confidence to carry on G: He couldn’t get over the first hurdle – he had a brick wall in front of him – he didn’t have the

confidence to stay with it. C: Its not just confidence – it’s the back up and the skills. G: Oh No – but he’s got the skills there – cause he did it for the first 2 months, then he had a

shower of rain or a flood or something and he couldn’t get over that next hurdle. C: But if he had had a 1:1 visit from an extension officer, perhaps a week before, the decision to

go back? G: But how are you going to know to do that? C: He needed support to do it. G: You have got to find out, he may have had an inability to speak within a discussion group as

well, and that would probably be the area that he needed to get support from. C: Or a mentor of some form. S: Or he may not have had the ability or even realise to say… G: I’ve heard that same thing more than once, I suppose that everyone else has heard it too:

when things get hard, people give up. C: When it gets hard you often go back to what you did before, I think this is why this is an

important story. G: I think people should know that these stories should be written, as they need to be heard. I

don’t think people do know. D: Jessica said at the start that the stories could be negative or positive. CM: But they are not necessarily negative stories – like it is bad for the farmer for the individual

farmer maybe. F: I think that some of the farmers read these stories and think that these are about me…[laughs] C: this is what I always thought about Target 10 – works in spring but not in summer.

Source: Transcript from the MSC process at a regional committee meeting

Text box 1 Segment of discussion at a regional committee meeting during story selection

The process of selection was similar at the CEC. However, at this forum the process tended to be

more formal and the stories were typed, titled, and had been allocated a domain. The stories were

circulated with the agenda before the meeting, along with a voting form (Appendix 9) to allow

participants to jot their comments down before coming to the meeting. Despite the fact that most

180

people had read the stories before the meeting, the committee still favoured the process of reading

the stories out aloud, in fact this became almost a ritual that people did not want to give up.

Feedback process

The various committees were required to document which stories they selected and what criteria

they used. The idea was that this information would be fed back to the project stakeholders on a

regular basis so that they could learn from the previous round of stories and interpretations. It was

intended that the monitoring system should take the form of a slow but extensive dialogue up and

down the project hierarchy each month (Davies 1996). It was unclear in Davies’ thesis how this

feedback process had occurred in Bangladesh, so again the process was arrived at through

experiential learning.

In the first three months I tape-recorded the entire story review session held at each of the regional

committee meetings. The discussion was then transcribed, and the reasons for selecting particular

stories were summarised and circulated in the minutes of the meeting. The idea was that the criteria

for selection of the stories would be appended to the story itself, so that the state executive

committee could consider the RC member’s interpretation of story, in addition to the story itself and

the storytellers’ interpretation. However, when the regional champions took over this process, they

found it to be too time consuming to tape and transcribe the whole meeting, so the main points were

jotted down and included in the minutes in bullet point form. Due to timing issues, it was

frequently not possible to attach this feedback to the selected stories that were sent to the state

executive committee meeting. However, feedback concerning the reason for selecting the stories

was generally verbally relayed by the respective regional co-ordinators who attended the CEC

meeting. We felt that this was an acceptable trade off.

However, at the CEC the feedback was much more systematic. The review session was tape

recorded at all five CEC over the 18-month period. The details of the selected stories and the

criteria by which they were chosen were systematically reported to all project staff and

stakeholders. This was conducted largely by means of an email list-serve immediately after the

meeting. In addition, the booklet Target 10 Evaluation Stories (Dart, 1999c) was produced

containing all the stories that had been selected by the CEC over the period of the year. Each story

was accompanied by the interpretation of the storyteller, comments from the CEC, and comments

from the purchasers. Thus the reader of the document could make a judgement on the story, and

also have access to information about how the project committees valued it, and the purchasers of

181

the project. The booklet also detailed an outline of the MSC process, and the findings of the

secondary analysis of the stories. Approximately 250 copies of the booklet were distributed to

project stakeholders.

Step 5 - Annual round table meeting

At the end of the 12-month trial period a round-table meeting was held with eight participants who

were considered to be ‘purchasers’ or ‘key influencers’ of the Target 10 Project. These participants

represented the Dairy Research and Development Cooperation (DRDC), NRE, the University of

Melbourne and the DRDC regional development boards. The round-table meeting took the form of

a facilitated group discussion in which all participants were asked to give their reaction, in general,

to the stories. They were then asked to nominate certain stories as being the most representative of

the sort of outcomes that they were interested in purchasing.

Step 6 - Secondary analysis of the stories

In total 134 stories were collected, transcribed and entered onto a database. On request of the

project, I conducted an additional step of analysing the stories en masse, the findings of which were

included in the booklet Target 10 Evaluation Stories (Dart, 1999c) and circulated to all

stakeholders. This analysis was done by examining the origin of the stories, the main themes, and

differences between the stories that were selected and those that were not. These findings are

detailed in the following section concerning the process outputs.

7.4 Analysis of Outputs

The 134 stories collected between May 1998 and May 1999 originated from all of the four regions

of Victoria where the project operates. These stories were written by staff from the Department of

Natural Resources and Environment (NRE), farmers, industry representatives and educators.

Twenty-four stories from the total were selected on the basis that they were considered by the

Central Executive Committee to be the most significant accounts of change in the specified

domains.

The story collection process was monitored and the stories were examined for overall trends in

content and origin. Firstly, the total group of 134 stories was considered. Secondly, the 24 stories

that were selected by the Central Executive Committee over the year along and the criteria with

182

which they were selected, were examined. Table 5 lists the questions that were used to structure the

following section and to describe the ‘results’ of monitoring the Story Process.

Table 5 Questions asked to help describe the stories

Data sets Qualitative questions Quantitative questions

Total 134 stories

that were collected

across the state

over the 12-month

period.

24 stories selected

from the total set by

the Central

Executive

Committee, and

scored by a group

of purchasers.

• What were the stories about?

• What unexpected farmer outcomes are described in

the stories?

• Did issues covered in the negative event stories get

acted upon?

• What is different about the 24 stories that were

selected and those that were not?

• What criteria did the project team use to select the

stories? What does this tell us about what the project

team values?

• What criteria did the purchasers use to judge the

stories? What does this tell us about what the

purchasers’ values?

• Did these stories change over the year as a result of

the feedback ie? Was there evidence of learning?

• How many stories

were collected and

when?

• Who were the

storytellers?

• Which programs

were mentioned in

the stories?

• Which domains of

change were

covered?

• What percentage of

stories concerned

negative news?

• To which level of

Bennett’s Hierarchy

do the stories

correspond?

A description of the stories

Number of collected during the process

Over the year, 134 stories were recorded and documented. The story review process has occurred at

five Target 10 Central Executive Committee meetings and over 10 regional committee meetings.

Table 6 presents the number of stories collected in each round. One round represents the time span

between two consecutive Central Executive Committee meetings. There is no clear increasing or

decreasing trend and in each round of the process between 22 and 32 stories were collected.

183

Table 6 Number of stories collected each round

Detail Total number of stories

Round 1 – May to July 1998 29

Round 2 – July to September 1998 26

Round 3 – September to November 1998 22

Round 4 – November 1998 to February 1999 25

Round 5 – February to May 1999 32

Total for 12 months 134

Origin of the stories

The three larger regions of the project all contributed approximately the same number of stories to

the process (see Table 7). The reason for the smaller number of stories originating from the North

East Region, is due to the fact that this region has very few staff. When the mean number of stories

per team member for each region is considered, it becomes apparent that there is little difference

between the four regions.

Table 7 Distribution of stories per region

Region from which the stories originated: Number of stories collected (n=124)

Mean number of Stories per staff

• The Northern Irrigation 38 3.5

• The South West 39 3.9

• Gippsland 46 3.8

• The North East 9 4.5

The majority of the stories were written by staff from NRE (who were mostly Target 10 extension

staff). However, of the selected stories, a higher percentage were written by farmers than in the

total group (see Table 8). Feedback from the Central Executive Committee explains that farmer

stories were perceived to be more powerful than stories written in the third person. It is also

interesting to note that three out of the five stories scored most highly by the purchasers were also

written by farmers. Clearly, stories told directly by farmers are valued by many of the project

stakeholders. This is reflected in the proportionally higher number of stories about farmers who

have attended multiple programs.

184

Table 8 Distribution of the story tellers

Detail Percent of total Percent of selected

Written by:

• NRE staff 77% 62%

• Farmers 13% 30%

• Other collaborators 10% 8%

Project programs covered in the stories

The stories cover all the programs, and 10% concern farmers who had attended multiple programs

(Table 9). However, there is a higher proportion of stories about farmers who had attended multiple

programs in the selected stories than from the total group. Feedback from the Central Executive

Committee suggested that stories that were about consolidating learning from the various different

programs were especially valued.

Table 9 Project ‘programs’ mentioned in the story

Detail Percent of total Percent of selected

Target 10 Program Involved:

More than one program 10% 20%

Discussion groups 15% 08%

Grazing Management Program 18% 17%

Nutrition Program 17% 13%

Dairy Business Focus Program 16% 21% Soils and Fertilisers Program 15% 08% Dairy Farm Performance Analysis 09% 13%

Subject material of the stories

En masse the stories present a picture of many farmers implementing part or all of the Target 10

message, and of farmers gaining from the programs in unexpected ways. Some of the common

themes running through the stories are issues such as feeling more in control of the business, feeling

empowered to challenge the consultant, gaining increased family communication after the

identification of mutual goals. There were also multiple stories about the far reaching benefits those

185

new farmers to the area or industry can gain through the Target 10 activities and networks. Table

10 lists some of the themes that have been the subject of more than two stories. The most frequent

theme described in the stories (11%) concerns how farmers experienced an increase in production

after adopting Target 10 practice.

Table 10 Description of themes that have been the subject of multiple stories

Main theme %

Change to Target 10 practice leads to increased productivity (pasture growth/utilisation, milk production or condition score)

11

Positive reaction to the Target 10 Project (useful information, unbiased information or meeting farmer needs)

9

The importance of discussion groups and sharing information 6

Change to Target 10 practice leads to saving money 6

A change of practice after making systematic decisions about operational issues rather than using rule of thumb

5

Empowered to make a strategic long-term decision 5

Identification of operational problem using skills acquired on program 5

Systematic operational decisions lead to feeling more in control of business 5

Empowered to deal with external experts through skill acquisition 5

Evidence of farmers taking up on-farm practices that were recommended by the project

5

Overcoming negative attitudes toward the project or project messages 4

New farmers to the industry gain valuable network through the project activities 3

Gaining a new understanding of the importance of the basic principles of the pasture management

3

The importance of learning to think rather than following a recipe approach 3

It is also clear from the stories that the storytellers value the concept of empowerment and of

farmers thinking for themselves rather than using a recipe book approach. In total, 34% of the

stories make some mention of increased control, independence, or empowerment.

186

Levels of Bennett’s Hierarchy to which the stories correspond

The stories cover a whole range of topics and subjects. To gain a picture of the spread, the stories

have been categorised into broad themes and arranged using Bennett’s Hierarchy of outcomes2 see

Table 12. However, it should be pointed out that each story may contain several themes, so the

stories have been categorised by the theme that rates highest in the hierarchy of outcomes. For

example, a story categorised as practice change will generally also contain themes such as changes

in knowledge and attitude which rank lower in Bennett’s Hierarchy of outcomes.

It is clear from Table 11 that stories that are situated higher in Bennett’s Hierarchy of outcomes are

more likely to be selected by the Central Executive Committee. The purchaser group also allocated

high scores to stories that contained themes relating to the higher levels of the hierarchy (Level 7 or

6). Thus, it can be inferred that most stakeholders of the project value the stories that are about

behaviour change and consequences of this change.

Table 11 Level of Bennett’s Hierarchy

Bennett’s level Percent of total Percent of selected 7 – Consequences 27% 50%

6 – Behaviour change 14% 21%

5 – KASA changes 35% 17%

4 – Reactions 15% 4%

2 Bennett describes a chain of events assumed to characterise most programs in extension (Bennett 1977). He

uses this chain of events to depict a hierarchy of objectives and evidence for program evaluation. Bennett lists

seven levels of goals in extension and claims that it becomes more difficult to evaluate at higher levels of the

hierarchy, as it becomes more difficult to show that changes at these levels are the result of extension activity

and not of other factors.

187

Table 12 Grouping of stories by theme using Bennett’s Hierarchy

Level of Hierarchy - Description of theme and number of stories belonging to each category (N=128) totalQuality of life increases

1 (1 selected)

Goals achieved in career steps

3 (2 selected)

New farmers to the industry gainvaluable network4

Level 7Consequencesof change

Improved pasturegrowth/consumption7 (1 selected)

Increased milkProduction6 (2 selected)

Production maintainedin harsh conditions1

Increased conditionscore of herd3 (1 selected)

Money is saved

8 (5 selected)35(12)

Level 6

Changebehaviour

Evidence of farmers takingup T10 recommended on-farm operational practices5

Empowered to make astrategic business decision

7 (2 selected)

Changed practice in terms ofcommunication within family

2 (1 selected)

Changed operational practiceafter systematic analysis offigures rather than rule ofthumb 5 (2 selected)

18(5)

Level 5 –KASA

Skills andempowerment

Identifyingoperationalproblem throughacquisition of newskills

6

Calculationsdoneindependently

3 (1 selected)

Calculationsbased on accuratefigures rather thanapproximation –lead tobreakthroughs4

Empowered todeal with externalexperts throughskills acquisition

6 (1 selected)

Systematicoperationaldecisions madelead to feelingmore in control ofbusiness6 (1 selected)

Women gainingmore recognitionin family throughskill acquisition

227(3)

Knowledgeandawareness

New KnowledgeBecomes incorporatedinto the prevalentculture2 (1 selected)

Gained a newunderstanding ofimportance pasturemanagement4

Gained deepertechnical knowledge ofoperational process

1

Gained deeperunderstandingstrategic thinking andbusiness planning2

Importance of learninghow to think ratherthan following recipe

413(1)

Attitude Overcoming negative attitudes toward the program2

Attitude towards an element of the Target 10 message3 5

Level 4Reactions

Project providesuseful, goodinformation4

Project providesunbiased advice

2

Project is meetingfarmers needs

2

Importance ofteam work

2

Importance ofon-farm days

2

Importance ofdiscussion groups

7- (1 selected)19(1)

Lessonslearned

FarmerOwnership

1

Confidentiality

1

Limits of theprogram

1

Needingfigures to solveproblem

1

What canhappen withonly partialunderstanding1

Comingunstuck inharsh seasonalconditions-2

The need forone to onesupport

18(2)

Other Evidence of the spread of Target 10 messages and practices to non-Target 10 participants3 3

188

Level 7 Consequences for the target group – 26% of stories

In total, 26% of the stories concerned Bennett’s higher level consequences. The largest proportion of

these stories concerned increases in production after adopting Target 10 practices. This was

expressed in terms of increased milk production, improved pasture growth, improved pasture

utilisation, or increases in the condition score of the herd. Several stories told of how farmers had

made financial savings after putting into practice knowledge that had been gained during Target 10

programs.

In addition to issues of productivity and profitability, there were some stories that related to overall

improvement in family quality of life, and the achievement of life or career goals. Another

unexpected consequence told in three stories was that the project had provided an invaluable social

and business network for farmers new to the industry or geographical area.

Level 6 Behavioural changes in the target group – 14% of stories

Stories categorised into Level 6 concerned behaviour change, but did not explicitly mention the

consequences of these changes. The changes described in this category ranged from operational

changes to long-term strategic changes based on considerable planning. There were also stories of

changed practice with regard to communication between family members.

Level 5 Changes in: knowledge, attitude, skills, motivation and group norms –

34% of stories

The most frequent themes in this category referred to issues such as empowerment and feeling more

in control of the farm business as a result of increased knowledge and skills. There were also several

stories which demonstrated skill acquisition in terms of how farmers had been able to solve technical

problems using skills that they had gained from attending Target 10 programs. Several stories were

about farmers gaining a much deeper understanding of why certain basic operational practices

(especially pasture management) are so important.

Level 4 The farmer’s opinion about extension activities - 14% of the stories

Stories that were not specifically about change, but concerned farmers’ positive opinions of project

activities fall into this category. It is striking that seven of these stories were independently written

on the importance of discussion groups – discussion groups are clearly a most valued event.

189

Other 9% of the stories

Some of the stories did not fit easily within the framework that was adapted from Bennett’s

Hierarchy. This was especially true of some of the stories about lessons learned. Three of the main

messages that came out of these stories were that:

• farmers may need more support in implementing new practices during difficult seasonal

conditions

• farmers with only partial knowledge of the Target 10 practices can run into serious problems

• farmers should be consulted before major changes to the programs are made.

Characteristics of stories selected by Central Executive Committee

The most frequent theme in the stories selected by the Central Executive Committee was that of

money being saved after a farmer adopts a recommended Target 10 practice (see Table 13). This is

reflected in the feedback that was given from the committee to the storytellers – that stories could be

improved by including some element of the ‘bottom line’ and should contain some tangible outcome.

The stories did change over the year and the regions obviously tried to take note of the feedback

given by the Executive Committee. After about six months, feedback indicated that novel stories

were more likely to be selected than stories that contained themes that had been heard before. Again,

this is illustrated by the fact that stories that had a unique theme, such as new knowledge becoming

incorporated into the prevalent culture, were selected despite the fact that they represent lower level

outcomes. Another factor that affected selection was that stories were more likely to be selected if

they clearly showed evidence that directly connected the occurrence of change to a Target 10

activity.

190

Table 13 Characteristics of stories selected by the Central Executive Committee (n=24)

Theme Level of Bennett’s Hierarchy

Number of stories selected

Percent selected from

sub-group Quality of life increases 7 1 100% Goals achieved in career steps 7 2 67% Change to Target 10 practice leads to saving money 7 5 63% Change to Target 10 practice leads to increased productivity (pasture growth/utilisation, milk production or condition score)

7 4 27%

Changed practice in terms of communication within family 6 1 50% A change of practice after making systematic decisions about operational issues rather than using rule of thumb

6 2 29%

Empowered to make a strategic long-term decision 6 2 28% New Knowledge Becomes incorporated into the prevalent culture

5 1 50%

Calculations done Independently

5 1 33%

Systematic operational decisions made lead to feeling more in control of business

5 1 17%

Empowered to deal with external experts through skills acquisition

5 1 17%

The importance of discussion groups 4 1 13% Lessons learned Other 2 22%

Feedback given by purchasers

Prior to the round table meeting, the purchasers were asked to score the stories individually (out of

10). When the scores for all 24 stories were examined, it became apparent that the purchasers-

respondents had very different reactions to the stories. Two stories were allocated the lowest score

by one purchaser, and the highest score by another! The purchaser group did not have a unified

vision as to what is desirable when given a choice of these stories. It was also apparent that most of

the stories were considered to have merit by at least one of the participants. This finding supports

the concept that evaluation is conducted in a value-pluralistic context; that the various stakeholders

hold differing values. Thus, negotiation and dialogue between the various evaluation stakeholders

(including between the purchaser and the provider) is essential.

There was general consensus at the meeting that overall the stories demonstrated: • Technical skill improvement after attending Target 10 programs.

191

• An array of unexpected positive outcomes achieved by the project and specifically with regard to

changes in attitude, achievement of personal goals, and changes in the farm business and family

situation.

Statements by two of the purchasers are provided in Text box 2 as examples of these comments.

I was taken by those stories that tended to indicate that not only had the program delivered the technical outcomes that are being sought by the industry, but also that, for a number of participants, it had brought about a change in attitude and achievement of personal goals as well.

Some of the description of the logic that flowed through stories (the technical logic) was very, very good and showed that these farmers in question had really grabbed this technology by the throat and were really milking it. The other sort of stories that got to me were the life-changing ones. And those two types of stories really got to me.

Source: Transcript from Round table meeting

Text box 2 Examples of comments from the purchasers concerning the stories

Why specific stories were valued highly by the purchasers

The discussion held at the round table meeting revealed that specific stories were highly valued by particular participants when: • the change described in the story was clearly attributable to the project

• the participants could relate to the story from their own experience

• the change described showed a diversity of learning

• the story demonstrated project reflection on negative events and subsequent learning

• the change described in the story demonstrated teamwork in action.

In most of the cases, there were considerable differences in interpretation of the stories; however, a

full consensus was achieved that the story ‘knowledge is power’ presented the most significant

account of change (see Text box 3). Because of this, considerable time was spent discussing why

this story was considered to be so significant. Comments from the meeting included that the story

represented the sort of outcomes that they were looking for from the project because:

• the event described in the story was the combination of the farmer having an articulated need and

the training program being available at the right time

• the program provided a stepping stone to another job and also into further education

• it shows a congruence of personal goals and business goals

192

• this story illustrates an example of how the program provided a springboard for farmers to go on

to what they want to do without prescribing a single pathway

• the farmer in this story has the right attitude

• how can we harness the learning from this story to learn how to encourage other young farmers to

have similar success?

The stories themselves

Texts boxes 3, 4, 5 and 6 provide four stories to illustrate points that have been made in the

discussion. The purchasers of the project gave Story 1 the highest score. The first story entitled

knowledge is power, was categorised as being concerned with ‘goal achieved in career steps’, an

outcome relating to Bennett’s level 7. This story concerned a farmer who had attended several

programs of the project. The feedback notes from the selection process often stated that the stories

written by farmers were more powerful, as they were first-hand accounts of farmer experience.

The second story entitled I’ll not be milking cows when I’m 55, was categorised as a story

concerning changes to the ‘quality of life’, an outcome relating to Bennett’s level 7. This story has

the characteristic conversational tone that was often present in stories that were recorded orally, and

were valued for this characteristic. It is notable that Stories 1 and 2 were written by farmers, and

were concerned with the highest level of Bennett’s Hierarchy, and that both were valued highly by

all the committees and staff.

However, the majority of stories in the process were written by staff, and were written in the second

person. Story 3, empowered, was written by a member of staff. Several stories with this theme of

‘empowerment’ were collected over the year, and this outcome was categorised as a Bennett’s level

6 outcome. This particular story generated lengthy discussion when it was reviewed. The outcome

of ‘empowerment’ is not highly related to the overarching production goal of the project – this is the

sort of story that ‘surprised’ some of the purchasers.

Story 4 entitled Saved me $2,000 is a concise story written by an extension officer and details short-

term savings gained from Target 10 activities. This story was particularly valued by the project as it

was a short story with tangible profit-based outcomes. Few stories were collected that concerned

long-term profitability or productivity increases, and stories collected under the domain of ‘changes

in profitability and productivity’ were generally short term in nature.

193

Selected at the round-table meeting as the most significant story

Story 1 Title Knowledge is Power Name of person recording story: MJ, Dairy farm employee Region: Gippsland Date of narration: Round 5 – 15 March 1999 Where did this happen: XXX When did it happen: 1996

What happened? In 1996, I was working on a 500-cow dairy farm and had no input to the day-to-day running. But I wanted to have more input. I decided to do something about it, so I attended the Target 10 Grazing Management Program followed by the Nutrition Program. After completing these courses, I had enough knowledge to start making decisions, such as measuring growth rates and adjusting rotation lengths. Gaining this knowledge led me to start a new job on a 550-cow dairy farm where I am presently managing all feed requirements.

I have been able to maximise pasture consumption to 10 t/ha; and, with 1 to 2 tonnes of grain fed, this will achieve a production of 1,000 kg of solids /ha on 850 mm of rainfall per year. I walk the farm weekly to measure growth rates, allowing me to adjust the rotation to suit the growth rate. I fill the gap with grain so as not to waste pasture. When harvest comes, I can use this method to cut maximum fodder without sacrificing the cows’ pasture.

Why do you think this is a significant change? These changes, which I have been able to apply to the day-to-day running of the farm, have made the farm more efficient. I am also a more efficient employee now, and I have seen what I can achieve by furthering my studies. Last year I completed a Diploma of Agriculture (dairy farm management), and this year I’m studying an Advanced Diploma (dairy farm management). My future plan is to manage the whole day-to-day running of a larger scale dairy farm in every aspect.

Feedback from the Central Executive Committee: • This story is a good example of one person going on to do more learning and expand his horizons. • It is good, as it is written by a farmer. • It is about building confidence, and the story even got down to changes in productivity. • It really shows the full picture.

Feedback from the Round-table Meeting: • Good positive story about practical learning. • Real change directly related to having done Target 10 programs (they look good in the resume)

with measurable results as well. • Substantial behaviour change as a result of the Target 10 program. But the profit/productivity

impact is not so clear. • Great story. Self-improvement outcomes, the subject has strong commitment to the industry and

the desire to achieve. These are the people that will make the industry move forward. • To me it is raw efficiency; that’s what really gets me. It is almost the most you would expect to get

from a program. The guy is really ready to roll and he has got the right attitude.

Text box 3 Story example 1

194

Title of story 2 I’ll Not Be Milking Cows When I Am 55 Name of person recording story: MS, dairy farmer Region Gippsland Date of narration: Round 2 – 21 August 1998 Who was involved: Farmer and family

What happened? We did the pilot Dairy Business Focus Program in March; and for the first time, my wife came along to something. We were able to look at our farm as a business, not just as a farm. As a consequence of doing the program, we did a few sums and made a few decisions. We worked out that we can afford to have her on the farm, and she has left her job at the bank. We will generate enough income on the farm to make it more profitable for her to be here. The kids will benefit from seeing her a lot more, and they won’t be in day care. So far this year, this has made the calving so much easier, we have a joint input, and it has been such a turn around in my lifestyle. It has been so good.

We actually went to the accountant yesterday to get some financial advice on how we should be investing off-farm. He was amazed that what we are doing is treating the farm as a business. I said: ‘Now everything that we earn on this farm is going to be put away so that I am not milking cows when I am 55 years old!’

We have got a debt-reduction program running for the next 12 months, but after that the money will be channelled to off-farm investment. I want to retire young enough to enjoy what we have been working towards for the last 20 or 30 years. My boss is 77 and is still working on the farm. If I am that fit when I am his age, I want to be touring around the world.

It has opened up our lives. We are now looking at off-farm investment, as capital investment on-farm is not that great. We are not going to invest in new machinery but are going to invest in contractors to do any work we can’t do. There is no point buying new machinery, as it depreciates. Instead, we will buy shares and invest off the farm. This proves that you can farm on 120 cows, you don’t have to get big, and you don’t have to milk a lot of cows. It just depends what you do with your money. If only we could educate the younger farmers to think ahead instead buying the largest SS Commodore or the latest dual cab. I followed the same track for a few years until we sat down and worked out where we were going and where we could be. We made a few mistakes in the past, but the past is the past.

Feedback from the Central Executive Committee: • This story generated lots of discussion. But is it really about profitability or quality of life or changes in farm practice? • The general consensus was that there needed to be more detail in the story for it to be about profitability. • It is a really powerful story that shows considerable change.

Feedback from the Round-table Meeting: • The story showed strong evidence of attitudinal change, leading to self-improvement and goal setting. These people will be high achievers and reap the rewards. They will be good role models for others who desire similar rewards. • This approach is okay, but it isn’t necessarily a prescription for others. • It has some good messages, but it hasn’t got all the answers. • This is a very good example of achieving the goal of the DBF Program: i.e., getting strategic thinking/planning followed by farmer action. • I liked this story as it highlights the diversity in personal goals and ways to get there.


195

Title of story 3: Empowered Name of person recording story: Julie Williams, Target 10 extension officer Region: Gippsland Date of narration: Round 5 – 1 April 1999 When did it happen: Between September and March 1999

What happened? At a review day of the Dairy Business Focus Program to share how people had gone with their planning, one of the participants, Barb, was very pleased with herself and feels that she is now more ‘empowered’. She has employed a consultant to help with decisions and bring new information into the family. Her husband grew up on a dairy farm and feels he knows best. The third person (the consultant) has helped her introduce new efficiencies and ideas.

She has recently organised her own holiday, something she’s never done before, and she’s making decisions for herself. Her husband has not supported her before, but now she believes she can do it. There is information available that she now has access to and is now confident to get on with it. She is also keen to do the DFPA Program, with a bit of help, to help with analysis of the farm costs and to help with decision- making. Why do you think this is a significant change? The interaction with the DBF group and the awareness the program created has empowered this woman to seek out assistance to be more proactive in her decision-making.

What difference will it make in the future? It will help her make more effective decisions in her farming business and her life generally.

Additional note from Gippsland Regional Committee: • I should add that Barb actually went on holiday on her own, but this is not a story about the family

breaking up. She got a lot out of the program and is now actively getting out and following some of the plans that she made during the DBF Program.

Feedback from Central Executive Committee: • This story shows a considerable degree of change has occurred due to the program. She was left

empowered to do things that she could not do previously. • I chose this story because of the magnitude of change. • Before it seems that the husband made most of the decisions; now there are more people making

decisions in the family. That’s a huge change.


196

Title of story 4: Saved Me $2,000! Name of person recording story: Frank Mickan, Target 10 extension officer Region: South West Date of narration: Round 1 – 10 June 1998 Who was involved: A farmer in the Soils and Fertilisers Program Where did this happen: Heywood When did it happen: April 1998

What happened? A pilot Soils and Fertilisers Program was presented at Heywood in South-West Victoria. Some calculations were done on day two; and by day three, a very ‘switched-on’ farmer had gone home and calculated the costs of nutrients that he had calculated that he needed for his farm. He had quotes from at least two fertiliser companies, and he calculated that a blend (from his own figures) would save him $2,000.

Why do you think this is a significant change? This man had obviously grasped the concepts of the calculations, shown by his ability to calculate the costing and levels of nutrients in the fertiliser blend.

What difference has it made already or will it make in the future? After two days of the program, this farmer had already saved himself $2,000 and will continue to do so in the future (although the amounts may vary). He can now check on the consultant’s recommendations.

Feedback from the Central Executive Committee: • I like it as it shows how a farmer applied what he had learned in a program to save him money. • It shows that the information that is given in the programs does work. • There is also an issue here about how the programs can improve the confidence of farmers and so

help them to help themselves (empowerment). • This story also illustrated how the farmer changed his behavior.


What the stories revealed in terms of evaluative information: my observations and analysis

En masse the stories present a picture of many farmers implementing part or all of the Target 10

message, and of farmers gaining from the programs in unexpected ways. The most frequent theme

for a story (and also the most frequently theme of selected stories) concern farmers who have

changed to Target 10 recommended practices and gained an increase in production. The second most

common theme for a story concerned farmers who had adopted of Target 10 practices and as a result

experienced short-term financial gains. These findings are consistent with the aims of the project,

and support the other evaluation findings (Target 10 Evaluation report 1999), that the Target 10

activities do have an impact on increased production and short term profitability.

197

However, in addition to stories concerning production and profit outcomes, many stories concerning

other types of change were collected and subsequently selected by the Target 10 committees. The

major themes running through these stories were that farmers, after involvement Target 10 programs,

felt:

• more in control of their business

• empowered to challenge the consultant

• that they had gained increased family communication after the identification of mutual goals

• empowered to make strategic long-term decisions

• able to make systematic decisions about operational issues rather than using rule of thumb

• able to achieve goals in career steps.

As the process progressed, we noted that the stories had improved and that the authors of the stories

were getting more skilled at knowing how to present the stories and what sorts of themes should be

presented in the stories. After six months, the stories seemed to be more ‘finished’ and to be more

closely related to change in the specified domains. The implication from this is that not only was the

project learning to run the process, but also the committee members (farmers, educators, industry

representatives, and NRE staff) were getting better at conceptualising impact.

The process of collecting and analysing stories has seen farmers, collaborators, and NRE staff sitting

together at committee meetings discussing and interpreting qualitative data, casting evaluative

judgements and negotiating about what constitutes a significant change. Feedback from the central

executive committee suggests that learning has also occurred in terms of increased skill in

conceptualising and capturing impact; over the year the storytellers became better at capturing

impact and responding to the suggestions that were provided in the feedback from the story review

process.

In total, 26% of the total stories collected deal with outcomes, both intended and unintended, that can

be classified as ‘the consequences of change’ (Bennett’s Level 7), which are typically found to be

hard to measure and difficult to attribute to project intervention though conventional evaluation

methods.

198

7.5 Problems During the Implementation

Competition

Early in the trial, one of the regional champions brought our attention to the fact that competition

amongst storytellers was becoming an issue in her region. At this stage staff were voting for stories

at the staff meeting. A situation occurred whereby three stories were selected by the staff team to go

to the regional committee, but by accident four stories were sent. The regional committee gave the

story that had slipped in by mistake the highest vote. This story also went on to be voted by the CEC.

Many people commented that it was a very good story and could not understand why it had not been

selected at the staff meeting. It turned out that certain people started voting for their friends’ stories,

at the expense of others. Things had become quite competitive, and people began to feel pressured.

Because of this, voting was subsequently confined to regional committees, and the CEC. At the staff

meetings, the stories were shared and commented upon, but no voting occurred.

Some regional committee members became very enthused when they were getting lots of ‘winning’

stories and disillusioned when their stories didn’t get accepted. The competitive factor was a problem

in a lot of ways. As an aside – my observations as a non-Australian, and having lived in both Hindu

and Muslim cultures are that under an Australian culture, people often dislike processes involving

comparison and promotion of individuals above others – and the MSC process is all about

comparison! This concept is colloquially named ‘the tall poppy syndrome’. It could well be that this

was not an issue in the Bangladesh where the approach was developed due to cultural differences.

Time

Participants frequently lamented that the biggest problem with the MSC process was the amount of

time that it consumed before and during meetings. Reading out the stories, discussing and voting for

them frequently took over an hour, and occurred at the majority of the meetings of the project. Some

felt that this time was justifiable in term of the benefits derived and others did not. This issue is taken

up further in the next chapter, in which an evaluation of the MSC model is presented.

199

Issues of culture and ethics

Early on in the process, it became clear that the approach advanced by Davies had not focused on

issues of ethics and confidentiality of the storytellers. There is a huge omission in the MSC process

described by Davies, in that it does not protect the identity of the informant. Perhaps it is telling of

the situation of farm families in Bangladesh, that confidentiality of the actors mentioned in the

stories was not mentioned as an issue. In fact, in the Bangladesh case the stories were seen to gain

credibility because they contained enough information for the storyteller to be located and the story

to be verified. When the approach was transposed to Australia, the confidentiality of the stories

became paramount. In the application of the MSC process to the Australian context a process was

needed:

• whereby people recording stories of their own experience gave and recorded their consent for

these stories to be used in evaluation

• whereby people recording stories that involved others, either render the stories anonymous, or

seek consent (and record this) from the subjects of the story

• for tracking whether the subjects have given their full consent

• to ensure that the stories are not used as evaluation material in a way which divorces them of

context

• for ensuring that stories are not published without going through the project committees and

gaining approval from the storyteller

• in which committees have some agreed rules and adhere to them as far as ethics are concerned.

In December 1998, protocols were set in place to deal with issues of confidentiality and ethics.

Firstly, a question regarding whether the subjects had granted permission for the story to be used for

evaluation was added to the proforma. Those stories that were placed in the booklet Target 10

Evaluation Stories were carefully dealt with, and all the subjects who featured in the stories were

contacted and their consent sought before distribution. However, there was still much debate about

how to protect the identity of subjects in the stories, especially stories that contained some element of

negative news.

During the 12-month trial, several people asked whether the stories could be used to place in

newsletters or to use as ‘good news stories to give to the minister’. From the beginning a rule was

200

made that no story could be published without the approval of the CEC. However, we continued to

debate whether it was ethical to use the stories for purposes other than evaluation. In the end, the

ethics code from Market Research Association was followed. This required that no stories collected

on the grounds of evaluation could then be used for other purposes. I suggest that if the subject

thinks they are giving information for one purpose it is unethical to use it for another.

7.6 Conclusions Concerning the Implementation of the

MSC Process

It was possible to implement the MSC process across a large Victorian extension project, albeit in a

slightly modified form than that advanced by Davies. A reference group appears to be an important

addition to the model, in order to guide the process, and allow it be adapted to the local context. To

successfully transpose the approach to a different culture (from a Bangladeshi savings and credit

project to Australian extension project) new processes were needed:

1. To get the ball rolling, some pre-implementation activities were needed. These ensured that

participants fully understood the process and were willing to become involved.

2. To manage the complex process of collection and selection of stories, people are needed to

facilitate and organise the process. One way of doing this was to appoint regional champions,

who together form a learning group for the process.

3. A process was needed to gain the consent of the storyteller, and to be able to track this consent.

4. In this Australian context, competitiveness and pressure became an issue that needed careful

management. Where possible an environment should be created to minimise unhelpful

competition by careful choice of the fora where stories are selected.

201

CHAPTER 8

THE IMPACT OF THE MSC PROCESS ON THE TARGET 10

DAIRY EXTENSION PROJECT

In this chapter the findings of a meta-evaluation of the MSC process that was implemented

across the Target 10 Dairy Extension Project are presented. Firstly, a model was constructed

of how the process was thought to bring about improvements to the project. From this model

nine ‘conjectured outcomes’ were identified including: increased participation, increased

knowledge, increased staff morale, and operational and strategic changes to extension

practice. Secondly, empirical evidence was used to test whether the conjectured had been

achieved, and if so, in what project contexts. Significant differences were found between

regions in the extent to which staff perceived some of the outcomes had been achieved. I

suggest that these differences were associated, amongst other things, with the structure and

nature of the meetings at which the stories were reviewed. Despite the differences in level of

success, the MSC process was able to meet many of the current evaluation needs of the

Target 10 stakeholders. The MSC process also appeared to complement the project’s existing

evaluation practice.

8.1 Introduction

In Chapter 7 the process of implementing the one-year trial of the Most Significant Change (MSC)

model was described and some of the outputs of this process were presented. To determine the

effectiveness of the MSC model it is important to look further than the outputs of the process, and

also to determine the value of the MSC model in general. To this end, a meta-evaluation study was

carried out to determine, firstly, the extent to which the MSC process was able to meet the case

project’s evaluation needs, and secondly, the potential of the MSC process to effectively meet

evaluation needs of extension projects more generally. The aim of this chapter is to present the first

part of the findings of this meta-evaluation. So here the emphasis is on investigating the extent to

202

which the process was valuable to the Target 10 Project and able to bring about project

improvements. In Chapter 9 an emphasis is placed on determining the value of the MSC model for

Australian agricultural extension projects in general.

In the first part of this chapter I focus on testing a series of conjectured CMO (context, mechanism,

and outcome) configurations (see Section 6.5) that together represent a theory for how the MSC

process brings about project improvements. In the second part of this chapter I focus on how well

the MSC process was able to meet the articulated evaluation needs and provide other valued

outcomes for the Target 10 Dairy Extension Project.

8.2 Testing the CMO Configurations for the MSC Process

In Section 6.5 an inductive process of developing key conjectured CMO (context, mechanism,

outcome) configurations for the MSC process was described. Nine CMO configurations were

identified, which represent testable hypotheses of how the MSC model brings about evaluative

activity and organisational learning and hence, project improvements (Table 14). In Section 8.4,

each of these nine conjectured CMO configurations are presented, and tested against empirical

evidence.

To test the conjectured CMO configurations (shown in Table 14), Pawson and Tilley’s (1997)

Realistic Evaluation model was followed in part, but not in entirety. A focus was placed on testing

the extent to which conjectured outcomes were achieved in different contexts. Less emphasis was

placed on testing whether the conjectured ‘mechanisms’ caused outcomes in specific contexts –

although mechanisms are referred to where the data existed. In each of the following sections that

refers to an individual CMO configuration, a figure is presented in which the mechanism is

presented in grey rather than black text – to signify the lower emphasis placed on mechanisms than

in the process advocated by Pawson and Tilley (1997). The mechanisms are also shown in grey text

in Table 14.

My rationale for placing less emphasis on mechanisms was related to the nature of the meta-

evaluation purpose, and the availability of data. The purpose of the meta-evaluation was not to test a

tight causal chain of events, but more to gain an idea of the sort of project contexts in which the

MSC process could contribute to project improvements. Pawson and Tilley stress the importance of

investigating how an intervention ‘works, and in what situations, and why’. In this meta-evaluation,

203

I was concerned with ‘how the MSC process worked, and in what situations’.

As explained in Section 6.5, a meta-evaluation questionnaire was developed for the purpose of

testing the relationship between the conjectured outcomes and different project contexts. The meta-

evaluation questionnaire (Appendix 2) was sent to all project staff and was returned by email. A

total of 28 people responded to the survey – a response rate of 80%. The survey was returned

anonymously, although most staff chose to state their names. However, it was possible to associate

names with most of the non-respondents in order to check the representativeness of the sample.

Five of the seven people who chose not to respond were very new staff who had not been involved

in the MSC process. Another person was away at the time of the survey. The non-respondents were

evenly spread across the four regions. Those who had been verbally antagonistic or pessimistic

towards the approach in the past did appear to be represented amongst the respondents. Therefore,

the people who responded to the survey appear to represent those most involved in the 12-month

implementation trial of the MSC model.

204

Table 14 Conjectured context, mechanism, outcome configurations for the MSC process

Context Mechanisms Outcomes

Those who participate are actively involved in the story review sessions.

People find the stories & interpretations interesting, engaging, and illuminating.

a) Increased attention paid to evaluative information

High attention to evaluative information: individuals are more likely to read/ pay attention, and remember the information.

Story review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.

Farmers and other committee members become actively involved in the evaluation process

b) Helps committee members to have a voice in the evaluation and to steer the project

As committees represent the wider farmer ‘voice’, this will help the project to continue to respond to farmer needs.


• The MSC process results in regular feedback with regard to the impact of extension activities.

• The competitive processes

whereby some stories (to which individuals feel ownership) are selected over others.

c) Increases morale

Reading / hearing success stories increases morale of staff and committee members. Having a story of significant change selected is also morale boosting for those who feel ownership over the events described in the story.


Regularly sharing and discussing the stories about outcomes that have occurred in farmers lives in relation to project activities.

d) Knowledge of impact

Participant’s gain new knowledge with regard to what has been achieved, what impacts are desirable.


Project stakeholders debate and reflect on project activities on a regular basis throughout the project.

e) Knowledge of each others’

values

Participants gain more knowledge of what the different stakeholders value.

205


Project stakeholders debate and reflect on project activities on a regular basis throughout the project. Inter-organisational reflection, communication and dialogue is enhanced.

f) A more fully shared vision

between all project stakeholders

Over time, the increased communication and dialogue will bring about a more shared construction of what has been achieved and what is desirable.

An organisational structure can accommodate change. That stakeholders are willing to enter into dialogue with regard to information gained from the process.

Feedback from the MSC process provides new knowledge with regards to the information requirements of different stakeholders.

g) Beneficial changes in the

way that the MSC process is run

Evolution and changes in the way the MSC process is run so that it can better meet the project’s needs.


• In their extension practice

staff try to reproduce outcomes deemed successful and avoid traps described in the stories of ‘lessons learned.’

• Decision-makers gain new

knowledge from the interpretation of stories and from the feedback from the purchasers. They may even examine specific stories to help in planning.

h) Beneficial operational

and strategic changes in practice

Extension staff make operational changes. Those involved in planning make strategic changes to the project through what they have learned in terms of evidence of impact, and stakeholder values of this impact. Purchasers make more informed funding decisions.


• Project members learn how to

better detect, reflect, and collectively respond to outcomes at the field level.

• Project members learn how to

better detect, reflect, and collectively respond to changes in stakeholders concerns and wishes with regard to the direction of the project.

i) Organisation develops a

better and more sustainable capacity to learn and change

By having a process in place to examine and challenge the value of what is occurring at the field level, the project team may gain a sustainable capacity to react to the changing external environment i.e., address farmer needs & purchaser requirements.

206

8.3 Different Contexts

The results of the meta-evaluation survey indicated strong patterns relating responses to regions.

Before examining the extent to which the conjectured outcomes were realised, contextual

differences between the four regions of the project will be outlined. The main quantifiable

differences concerned the number of staff attending the respective regional committees (Table 15),

the number of farmers and other people who attended the committees (Table 16) and the number of

stories contributed by people other than the project staff (Table 17).

Table 15 Staff attending their regional committee meetings during the trial

Regional location of respondent Northern Irrigation

Region South West

Region Gippsland

Region North East

Region 2/8

(25%) 7/7

(100%) 7/11

(65%) 2/2

(100%)

Table 16 Attendance at regional committee meetings during trial

Northern Irrigation Region

South West

Region

Gippsland Region

North East Region

Number of staff 2-3 6-7 4-6 2 Number of farmers 4-6 2-6 4-6 6-8 Number of other nominated committee members

1-3 7-14 1-2 1

Total 7-12 15-27 9-14 9-11

Table 17 Number of stories contributed by farmers and other non-staff committee members per

region

Region Number of stories by farmers

Number of stories from

other non-staff committee members

Total number of stories

Northern Irrigation Region

1 4 38

South West Region 4 4 39

Gippsland Region 13 3 46

North East Region 0 2 9

207

The Northern Irrigation Region

During the 12-month trial of the MSC model, staff from the Northern Irrigation Region attended

fewer regional committee meetings than staff from other regions. The practice was for between one

and two members of staff to attend the regional committee meetings which farmers attend. The

regional structure resulted in fewer staff attending regional committee meetings than in other

regions (see Table 15). This was the only region where stories were selected at the staff meetings.

Therefore, extension staff experience of the process was largely associated with discussing the

stories among staff members rather than committee members. In this region staff collected all but

one story, which was collected by a committee member (see Table 17). During the 12-month trial

there was also considerable staff turnover in this region.

The South West Region

The South West regional committee meeting is large, and all staff are expected to attend (see Table

15). In addition to extension staff, other staff from NRE are also members of the committee (largely

those working in dairy research). At the time that the evaluation was conducted, the committee

membership was 41 and was made up of:

• 10 farmer members with between two to six regular attendees

• 12 industry representative members with around eight regular attendees

• five educator members with two regular attendees

• 14 NRE staff members with seven regular attendees (who are the extension officers).

Up to half of the committee members attended the meetings, with some people attending

sporadically. These regional meetings were larger and more formal than regional committee

meetings in the other regions. The meetings also tended to be lengthy and have a full agenda.

The Gippsland Region

Gippsland has a very different committee structure to other regions, with three sub-regional

committees in addition to the regional committee. There was a high representation of farmers at the

regional committee, and lower representation of staff (see Table 16). The other staff attend the sub-

regional meetings, where farmer participation is also high. The meetings tend to be informal in

structure and appear to have relatively regular attendance. The Gippsland Region contributed

considerably more stories authored directly by farmers than the other regions (see Table 17).

208

Purchasers and the Central Executive Committee (CEC) members explained that stories authored

directly by farmers were particularly powerful. In confirmation of this claim, four of the five stories

selected by the purchasers originated from the Gippsland Region.

The North East Region

The North East Region is small, with only two extension staff. Regional committees are held less

frequently than in the other regions (every six months, rather than every 2-3 months). These

committees are attended by several farmers and are relatively informal.

A scale for different meeting contexts

In was hypothesised that the achievement of many of the positive outcomes from the MSC process

was dependent on the atmosphere and structure of the meetings where the stories were reviewed.

For this reason, the regions were placed into a continuum regarding the extent to which the

meetings (at which the stories were reviewed) provided an environment conducive to discussion,

reflection and exchange of views between different stakeholders (see Table 18). Each region was

allocated a score, which was used in quantitative analysis to test whether the structure and

atmosphere of meetings related to staff perceptions of extent to which beneficial outcomes were

achieved.

Table 18 Extent to which meetings in the different regions provided a context conducive to

discussion and exchange of views

South West Region Northern Irrigation Region

Gippsland Region

North East Region Size of meeting Large

(15-27) Small (7-12)

Small (8-13)

Ratio of farmers to other attendees

Low (between 1:4 and 1:7)

High (approx. 1:1)

High (approx. 1:1)

Level of exchange between farmers and other staff

Low (as the meetings are so

large)

Low (as few of the staff

attend)

High

Level of informality of meetings

Low Medium High

Score for environment conducive to open discussion and exchange of views

Low

1Medium

2High

3

209

8.4 Findings

The extent to which the conjectured outcomes were achieved in the different contexts is examined

in the following sections. The score allocated in Table 18 was used as a ‘proxy indicator’ for

context, and the findings were therefore examined for differences between the three categories of

context (concerning the structure and atmosphere of meetings where the stories were reviewed). In

some cases other variables were also used to investigate the different responses, including the

number of story review sessions the individual had attended, the number of stories read by

individuals and the extent to which the stories were remembered. The quantitative data was drawn

largely from the staff questionnaire, with supplementary data being used to help explain the

findings. The supplementary data came from a wide range of sources including face-to-face

interviews and focus group discussion, open questions in the questionnaire, and my observations of

the implementation of the MSC process.

a) Increased attention paid to evaluative information

________________________________________________________________________________

The first conjectured CMO configuration is shown in the flow chart above. It is hypothesised that

when people actively participate in the story review sessions then they will find the stories and

interpretations interesting and so will pay considerable attention to the evaluative information. This

hypothesis was tested by examining the relationship between the conjectured outcome and context.

The context was determined by the extent to which individuals had attended the story review

sessions. The attainment of the outcome of high attention paid to evaluative information was

judged by examining the extent to which individuals had read the stories, and to which they

remembered them. For this CMO configuration, the conjectured mechanism (that people find the

stories interesting) was also examined, but it was not tested against the other variables.

ContextThose who participate are actively involved in the story review sessions.

MechanismPeople find the stories and interpretations interesting.

Outcome High attention paid to evaluative information: • individuals are more

likely to read the stories • individuals are more

likely to remember the information.

210

Conjectured mechanism – interest in the stories

In the questionnaire, respondents were asked to openly comment on their impression of the stories

in general, and the responses were coded into a number of categories and presented in Table 19.

Responses were all generally positive; with many respondents making some reference to the fact

that the stories were interesting or enjoyable (58%); one respondent stated that they were ‘fantastic,

and inspiring to read’. In addition to the comments in the Table 19, two respondents stated that they

found the interpretation of the stories to be more valuable than the stories alone.

Table 19 Respondents’ descriptions of the stories

Main words to used to describe the stories % of responses

n=28* Enjoyable or interesting 54% Capture the impact of the project well 32% Inspiring or encouraging to the reader 18% ‘Real world’ feel or a ‘human’ approach 18% Powerful 11% Good approach 7% Good in that they illustrated different peoples views’ on the project 7% Easy to read 4% Important for advocacy 4% *One respondent chose not to answer this question and several respondents gave more than one response

However, one respondent stated that at the beginning of the 12-month trial the stories were not

meeting his expectations:

…they were a bit corny – a bit ‘Micky Mouse’ – I was not quite sure how these were going to go. So I

suppose I was more skeptical at the start’ (quote: project staff).

Feedback from the story review sessions at the CEC meeting stated that committee members felt

that the stories became better and more credible as the process ran, and as people learned what sort

of stories were valued (see Section 7.4). From the data it appears that that majority of staff did find

the stories interesting, and perceived them to improve over time. The demand for additional copies

of the ‘story booklet’ bears witness to this, with over 300 copies now having been circulated. The

interest in the stories (and interpretations) clearly extended beyond those immediately involved in

their collection and selection.

211

Conjectured outcome – attention to evaluative information

All respondents reported that they had received the booklet entitled Target 10 Evaluation Stories

(Dart, 1999c). This contained the 24 stories selected by the CEC over the year, and feedback from

the purchasers concerning these stories.

The extent to which individual staff had read the stories was examined against the number of story

sessions they had attended, but no significant correlation was found between these variables (Table

20). The two respondents who reported that they had not read any of the stories in the booklet, both

stated that they had listened to some of the stories reviewed at meetings. It can therefore be

surmised that all respondents were exposed to at least some of the stories.

Table 20 Number of stories read in booklet against number of story sessions attended

Number of stories read in the booklet Number of sessions attended

none a few most all Totals

0 0 0 0 0 0

1 0 1 2 0 3

2-3 1 2 3 1 7

4 or more 1 6 7 3 17

Totals 2 9 13

4 n=27

Gamma value =0.063, approximate significance =0.814 No statistically significant differences at p<0.01.

Respondents were also asked to comment on their ability to remember the stories. Their responses

were coded into eight categories and are displayed in Table 21. The majority of respondents

remembered at least some of the stories. There was no statistically significant relationship between

individuals’ memory of stories and the number of story review sessions attended, or with meetings

to which they attended.

212

Table 21 Extent to which respondents remembered the stories

Memory of the stories* Number of sessions attended

1 2 3 4 5 6 7 8 Totals

0 0 0 0 0 0 0 0 0 0

1 0 0 0 0 1 1 0 1 3

2-3 0 0 0 1 4 0 0 2 7

4 or more 1 1 1 3 3 6 1 0 17

Totals 1 1 1 4 8 7 1 3 n=26**

Gamma value = -0.12, approximate significance =0.62. There was no statistically significant difference at p<0.01. * Memory of stories was scored from 1, (remember none) to 8 (remember all the stories). **There were 2 non-

respondents

While only 15% of the respondents stated that they remembered many of the stories well, it seems

that many of the respondents suggested their own ‘mechanisms’ for why they recalled particular

stories or parts of stories quite vividly. The reasons given for a particular story ‘standing out’ were

variable and included:

• that the respondent had written the story

• that the respondent knew the farmer referred to in the story

• that the respondent had experienced a similar event as described in the story

• that the particular stories were discussed at length at a meeting that the respondent had attended

• that the story had been selected at the CEC

• that the story was particularly ‘powerful’ in content.

In other cases, respondents remembered parts of stories; for example, one person remembered the

titles or the punch lines of the stories. Several people commented that the catchy ‘titles’ had also

helped them to remember the stories.

One respondent commented that an important beneficial outcome of the MSC process was that

many staff remembered ‘common stories’ and that this may have augmented the collective memory

of the organisation:

The fact that the stories contribute to the collective memory is a very important thing – considering

the really high turn over of staff in the project (quote: project staff).

213

There are two important points behind the concept of a ‘collective organisational memory’. Firstly,

having the stories documented and available means that new staff can tap into the past experience

and achievements of the project and what the stakeholders value. Thus, as old staff leave and new

staff enter the project, some of the collective experience is retained. Secondly, as many staff and

project stakeholders hold common stories and interpretations in their memories, this may contribute

to the creation of a shared vision as to what the project has achieved and how these achievements

are valued. This concept is further elaborated in Section 8.4f.

Relationship between the conjectured context and outcome

There was no obvious pattern relating the conjectured context (degree of participation in the

process) and outcome (attention paid to the evaluative information). In this case the conjectured

mechanism (interest in the stories) and the conjectured outcome (high attention to evaluative

information) seem to have been realised regardless of the context.

In all regions, and at all levels of involvement, people found the stories interesting, and read a

considerable number of them, and therefore paid considerable attention to the evaluative

information created through the MSC process. While most people seemed to remember parts or all

of some stories, the extent to which respondents stated that they could remember the stories varied

widely from person to person. It could be that people in general (regardless of context) find stories

interesting. This alternative hypothesis would concur with the sensemaking literature (e.g., Weick,

1995) that suggests that stories are the preferred sensemaking media in organisations.

b) Helps steering committees to ‘steer’ the project

________________________________________________________________________________

The second conjectured CMO configuration is shown in the flow chart above. One of the ideas from

the evaluation theory model was that the MSC process would help the committees have a ‘voice’ in

the evaluation, and help the committees to function well and to steer the project. This question was

posed to staff in the meta-evaluation questionnaire. This conjectured outcome is congruent with the

ContextStory review occurs in meetings that have a structure and atmosphere conducive to open discussion and exchange of views between staff and farmers.

MechanismFarmers and other stakeholder become actively involved in the evaluation process.

OutcomeThis helps the farmers to have a voice in the evaluation and help steer the project.

214

underlying conceptual framework of the project (Chapter 6, Figure 13). One of the facets of the

Target 10 conceptual framework is that ‘at the community level, the project provides opportunities

for stakeholders to participate in the development, organisation, and delivery of project elements, as

well as in communication strategies and collaborative alliances at state and regional levels’

(Boomsma et al., 1996).

Conjectured outcome – MSC process helps the committees to steer the project

Table 22 shows that over half of the respondents felt that the process had helped the committees

steer the project ‘a lot’ or ‘to some extent’. Nevertheless, some respondents felt that it had no effect

or negative effect.

Table 22 Staff perception of the extent to which the MSC process helped farmer committees to

‘steer’ the project against meeting context

Meeting context *

Response 1

South West2

Northern Irrigation

3Gippsland and

North East

Totals

Hindered a lot 0 0 0 0

Hindered to some extent 1 0 0 1

No effect 2 2 2 6

Helped to some extent 4 3 4 11

Helped a lot 0 2 6 8

Totals 7 7 12 **n=26 Gamma value =0.556, approximate significance =0.008, Statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** 2 people chose not to respond to this question

However, there were statistically significant differences between the three meeting contexts. This

can be explained by examining qualitative data. In the South West participants found the process

less enjoyable and excessively time consuming and indicated that MSC had not helped their

regional committee to steer the project. Some participants explained that this could be due to

problems with the meetings themselves. Informants attributed difficulties with the MSC process to

the fact that the stories were seen as an extra task in an already busy timetable at the regional

committee meeting. A distinct concern was raised that the meetings were too large for everyone to

215

have their voice heard:

I think the pressure comes from the size of the group, because if you have a big group there is always

pressure for everyone to have their say. The meetings just drag on and on for that very reason, that

everyone wants to have their say and everyone wants to know what got discussed at the last meeting –

it could be a function of that. We are pressured for time in the Target 10 meeting. There’s another

agenda too here – and that’s quite a few of us in the SW feel that you go to meeting after meeting, and

we’d like to cut the Target 10 regional committee out. You have to bring a note from home if you

don’t go to the meeting! (Quote: extension staff from the South West).

In contrast, in Gippsland the MSC process was perceived as being highly beneficial to the work of

the regional committees. Here meetings are smaller, with between 8-15 people attending. They are

also more informal in structure and appear to have a more regular attendance than in the South

West. One respondent explained that the MSC process helped the regional committees to feel they

were contributing to the project:

One the things that we have had increasing difficulty with has been the roles of the regional

committees. Back at the start there was a whole heap of work for them to do in terms of developing

the content of the programs, organising launches and many more things. Over time the role of the

committee has changed – and there are not the same opportunities for the committees any more. In

the Gippsland Region the committee got a lot of enjoyment and satisfaction out of the story process –

and felt they were actually making a contribution to the project. They saw this more as the evaluation

than the other processes – such as taking attendance records at courses and identifying whether

learning objectives had been achieved and whether competencies had been achieved. Because the

committees were directly involved this – they don’t get involved in the rest of the evaluation (quote:

extension staff).

A recent report (Dart, 1999b) suggests that when program development is complete (as was the case

at the time of the 12-month trail of the MSC process), the role of the regional committee members

change. The role of farmers and industry representatives becomes one of overseeing project

activities, and they assume a more passive role in the committee structure. In this role, farmers have

less opportunity to influence the project and experience less satisfaction and sense of ownership.

Thus, the MSC process was implemented at a particularly opportune time, when there was less

opportunity for farmers to be involved, yet still a strong commitment to have a strong farmer voice

in the running of the project.

At the Central Executive Committee (CEC) meeting, the story review process was well received.

216

Participants who attended the CEC meetings suggested that the MSC process had been particularly

effective in encouraging communication in the meetings, especially in regard to farmers’

involvement in this dialogue:

I think there has been an increased communication between and within the committees as people have

talked much more openly. The stories have become a vehicle and an excuse for people to talk more

openly. It has been a way that people have increased their confidence to be heard at meetings –

because the stories are a subject matter which farmers are very familiar with. Management meetings

can often become abstract and people; especially farmers sometimes feel "do I have anything to

contribute to this?" But when you talk to farmers about something in the stories then you are talking

about something that is very familiar and dear to them. So they are much more relaxed and confident

when they are talking (quote: one of the project co-ordinators who attended all CEC meetings).


While no firm cause and effect claims can be made, the outcome (the MSC process helps committee

members ‘steer’ the project) does seem to be more likely to occur in the conjectured context (where

there stories are reviewed in meetings more conducive to open discussion and exchange of views

between staff and farmers). The conjectured context-outcome configuration does appear to offer a

plausible explanation for the data. The MSC process seems to have helped committees to ‘steer’

the project in contexts where there are small meetings with high levels of farmer participation.

Farmer involvement in Target 10 committees is seen as a crucial part of the conceptual framework

of the project; involving farmers in steering committees helps the project to be focused on farmer

needs, and helps build a community supportive of change. An unanticipated outcome of the MSC

process was that in some contexts it provided a role for these committees, at a time when their role

was uncertain.

217

c) Increased morale

________________________________________________________________________________

A conjectured outcome of the process was that hearing or reading ‘success stories’ would be

motivating for those who feel some ownership of the stories. A second associated outcome is that

having stories selected would also be motivating for those who feel some ownership over the

‘winning’ story. To test this conjectured outcome, staff were asked (through the use of the

questionnaire) whether the stories had helped staff morale. Following this, the qualitative and

observational data were examined for evidence about whether the competitive selection (at both

individual and region levels) had affected staff morale.

Conjectured outcome (1) – increased morale through hearing stories of

change

For some staff, the MSC process boosted their morale, especially through hearing how their work

had contributed to positive outcomes in farmers’ lives. Table 23 shows how 22/26 extension staff

stated that the approach had helped staff motivation ‘a lot’ or to ‘some extent’.

The data in Table 23 suggest that there were regional differences (although not statistically different

at p< 0.001). Staff from the South West Region were less likely to allocate a high scores for

motivation than the respondents from the other regions. One respondent from this region stated that

the process had actually hindered staff morale. This can be partly explained by examining the

qualitative data. Staff from the South West Region stated that they felt pressured into generating

stories and found this to be uncomfortable. These sentiments are illustrated through the extract

presented in Text box 7.


Mechanism Regular feedback occurs with regard to the impact of extension activities.

The competitive process is motivating for those concerned with the ‘winning’ story.

OutcomeIncreased morale by: 1) reading / hearing success stories 2) having a story of significant change selected.

218

Table 23 Staff perception of morale boosting effect of the MSC process against meeting context

Meeting context *

Response 1

South West2

Northern Irrigation

3Gippsland and

North East

Totals



No effect 2 1 0 3


Helped a lot 1 3 5 9Totals 6 8 12 **n=26

Gamma value =0.500, approximate significance =0.057, Not statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 2 non respondents

C.I: I thought we had organised it that everyone should write a story and bring it in? M.N: Yes that’s what happened, they did do this but they felt very pressured, and they were very

upset about being pressured. One meeting that Jessica wasn’t there, when Ian [regional champion] wasn’t there, Jessica was sick.

DJ: There was quite a lot of confusion at that meeting wasn’t there? MN: What it came down to was that a lot of people had been doing course work and this was… C.I: An extra thing on the top? M.N: It was… but when it really came down to it they were not getting out to the farmers to

actually pick up the stories. C.I: So it was the extension staff that got upset? M.N: Well it was, but they felt like they were under pressure, but when it got down to the bare

bones of it they weren’t having enough contact with farmers. C.I: The contact was there to be able to generate the stories. M.N: They were under pressure and not able to do their 1:1 farm visits and consequently the

stories weren’t coming through. And the rest of the committee weren’t pulling their weight and there were other people that perhaps had more farmer interaction than they did and they were just sitting back expecting them [extension officers] to come up with the stories. So we had that out, and then we agreed once again, and that’s when the farmers decided to come into the process.

Text box 7 Extract of discussion between two extension staff from the South West Region, taken

from a focus group discussion between regional champions

219

In the particular year of the trial, the staff (and committee members) from the South West Region

were busy with organising a large conference in their region, and perhaps had more scheduled work

than extension officers in other regions. One informant explained that it was hard to collect stories,

as she was not having any contact with farmers during that particular period. So a combination of a

busy schedule, low contact hours with farmers, and pressure from ‘above’ to produce stories,

appears to have minimised the impact of the morale boosting effect of the MSC process in this

region. This case also illustrates the point that the story collection process should evolve naturally

and people should not be forced into participating against their will.

Conjectured outcome (2) – competitive aspects of the MSC process

A second conjectured outcome thought to affect staff morale is the competitive aspect of the story

selection process. Observation and qualitative data revealed that some staff became particularly

enthused when the stories, for which that they felt some ownership, were selected. In these cases,

the language was frequently of ‘winning’ or ‘losing’. This mechanism can have two possible

outcomes; those who ‘win’ feel increasingly enthused, and those who ‘lose’ become increasingly

demoralised, or negative toward the approach. This mechanism appeared to occur at two levels:

between the regions, and between individual staff within one region.

With regard to the regional level, it seemed that there was some competition between the different

regions to ‘win’ stories at the CEC meeting. Records show that ‘wins’ were approximately evenly

distributed across the three major regions. Nevertheless, voting for the most significant stories at the

state and purchaser level did appear to create competitive feelings between the three large regions.

The competitive aspect between different regions is illustrated by the comments presented in Text

box 8.

I have not taken the feed back session very seriously, as often it seemed to be a point scoring exercise against the other regions, to see how many stories ‘got up’ from our region (quote: extension staff).

Another important thing said by SP was that she was getting really pissed off with being compared to the other regions (quote: extension staff).

The feelings I have in the state exec meetings? Its them and us – its very difficult to take that hat off and see which is the best story. Because I want the stories from my region to win! Luckily I don’t think the comments that I make have as much impact as when farmers make comments about the stories (quote: extension staff).

Text box 8 Comments concerning the competitive aspect of the MSC process

220

At the individual level, some respondents also explained that they experienced competitive

pressure. In the Northern Irrigation Region, individual competition was particularly problematic.

Initially in this region stories were reviewed at monthly staff meetings. This was the only forum at

which staff effectively voted for stories written by themselves. This process was not seen as

successful and was eventually changed in line with the other regions. The prime reason for

changing this structure was that the voting process between staff had become excessively

competitive, with people voting stories submitted by their ‘friends’ and voting against stories

written by people from other factions of the group (see Section 7.3). Nevertheless, the competitive

aspect of the process did not appear to effect staff perception that the process had been beneficial in

terms of staff morale. The main issue that people were concerned about was that competitiveness

would lead to bias in the story selection process.


For this CMO, the conjectured outcome (increased motivation) did seem to be realised in the

conjectured context (where stories were reviewed in meetings that were conducive to open

discussion and exchange of views between staff and farmers). The CMO configuration did appear

to be a plausible explanation for the data.

However, there appeared to be several additional contextual factors important to bring about

increases in morale that were not anticipated. It seems important that participants enter the process

on a voluntary basis. The story selection process also involves a considerable amount of

competition, and this can be either motivating or de-motivating for those involved – this being

partly determined by whether the stories, for which the individual feels ownership, are selected by

review panels. It also appears that unhelpful competition can be reduced by ensuring that groups of

people are not casting judgement on stories authored by themselves.

221

d) Increased understanding of the project’s impact

________________________________________________________________________________

This conjectured CMO is represented in the flow chart above. The context-outcome relationship

was tested by examining informant’s perceptions of increased staff understanding of project impact

across the different regional contexts.

Conjectured outcome – increased understanding of project’s impact

My observations of the implementation of the story review sessions were that committee members

were ‘surprised’ at some of the stories – they were hearing stories that they did not already know.

The staff questionnaire also supported this conjectured outcome; with all the respondents stating

that the approach had helped staff understanding of project impact ‘to some extent’ or a ‘lot ‘(see

Table 24). This was the highest scoring of any of the variables considered, with no respondents

indicating the MSC had no effect on increasing their understanding of the project’s impact.

Nevertheless, there were statistically different responses across the three regions. Unlike

respondents from other regions, non of the respondent’s from the South West Region stated that the

MSC process had helped staff understanding ‘a lot’. This supports the conjectured context-outcome

configuration that learning from the stories is more likely to occur in small informal meetings where

there is opportunity for staff and committee members to share their views.

My observation of the process in all regions was that less time was spent debating the stories in the

South West than in the other regions. It could be said that the large meetings and formal atmosphere

of the South West meetings was less conducive to learning. Other plausible explanations for the

regional differences are concerned with the number of stories generated, and the degree of farmer

involvement in the story collection and selection process. Perhaps hearing stories directly from

farmer-clients is more likely to bring about new knowledge than hearing stories told by extension

staff – there was a lower ratio of farmers to other committee members at the South West regional


Mechanism Regularly sharing and discussing the stories about outcomes that have occurred in farmers lives in relation to project activities.

Outcome Participants gain new knowledge about achievements, and impacts considered desirable.

222

committee meeting than in other regions (see Table 16).

Table 24 Staff perceptions of the extent to which the MSC process affected staff understanding of

project impact against meeting context

Meeting context *

Response 1

South West2

Northern Irrigation

3Gippsland and

North East

Totals



No effect 0 0 0 0



Totals 7 8 11 **n=26 Gamma value =0.813, approximate significance =0.0007, Statistically significant difference at p<0.001. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 2 non-respondents

At the round table meeting several of the purchasers also expressed the view that the approach had

helped them to further understand the impact of the approach:

I found the stories really illuminating, I enjoyed reading them. I felt a number of them really nicely

demonstrated the technical competency, or skills learning outcomes of Target 10. And I thought there

was some other stories in there, which to me showed how participating in Target 10 had actually

changed people’s attitude to learning, and fact in some cases it had actually affected the farm business

and family situation! (Quote: Purchaser at the round table meeting).


All of the respondents perceived that the conjectured outcome of increased understanding of project

impact had been achieved to some extent, regardless of regional context. However, there did appear

to be some relationship between the strength of the perceived achievement of the outcome and the

conjectured context. Respondents who attended meetings that were more conducive to open

discussion and exchange of views between farmers and staff were more likely than other staff to

perceive that the MSC process had helped staff understanding ‘a lot’.

223

e) Increased knowledge of each others’ values

________________________________________________________________________________

Another conjectured outcome was that MSC process would bring about increased knowledge of

other stakeholder values through the regular feedback from the interpretation and selection of

stories. The idea was that this outcome was more likely to be realised in meeting contexts more

conducive to open discussion. To test this conjecture, staff were asked whether they felt that the

feedback from the story review sessions had (1) helped them to understand what outcomes were

valued by the CEC and (2) helped them to understand what outcomes were valued by the purchasers

of the project. These outcomes were examined in relation to the different regional contexts.

Conjectured outcome (1) – Increased knowledge of the CEC’s values

Table 25 shows that the majority (20/25) of informants stated that the approach had helped staff to

understand what outcomes the CEC valued. Yet five were of the opinion that the feedback had had

no effect.

There were no statistically significant differences between respondents from the different regions.

However, unlike respondents from other regions, none of the respondents from the Northern

Irrigation Region stated that the MSC process helped them ‘a lot’ to understand what outcomes the

Central Executive Committee (CEC) valued. The comments relating to this question indicated that

staff from this region felt that they did not get sufficient positive feedback or involvement with their

regional committee, or from the CEC. Staff in the Northern Irrigation Region attended significantly

less regional committee meetings than those from other regions (see Table 15). These staff had

much less first hand experience of sessions where the regional committee discussed and selected

stories.


Mechanism Project stakeholders debate and reflect on project activities on a regular basis throughout the project.

OutcomeParticipants gain more knowledge of what the different stakeholders value.

224

Table 25 Extent to which the MSC process affected staff understanding of outcomes valued by

the CEC against meeting context

Meeting context *

Response 1

South West2

Northern Irrigation

3Gippsland and

North East

Totals



No effect 1 3 1 5



Totals 7 8 10 **n=25 Gamma value =0.25, approximate significance =0.35. Not statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 3 non-respondents

In addition, there was a tendency for the Northern Irrigation regional committee to be quite critical

of the stories that were submitted. On one occasion they sent the majority of stories back to the staff

to be re-written:

At the regional meeting they read the four stories through and for the first time they were introduced to

the feedback from the state executive meeting. And they took this very seriously and actually rejected

three of the four stories saying that they didn’t have enough detail and that they were too subjective!

So they had to be re-written, and these stories have already been re-written at the dairy extension

meeting. They seemed to be going for perfection and I was told that it was a fairly tense meeting

anyway. But it seems like they did discuss the stories anyway (quote: staff member from the Northern

Irrigation Region).

Some respondents from this region also explained that they felt the feedback process from both the

regional committee and the CEC had not been fully effective.

It appears that for most staff, the story process helped them gain a better understanding of the

outcomes valued by the Central Executive Committee. However, this outcome can only occur

where there is regular (and constructive) feedback from the story selection process at both the

regional and state levels.

225

Conjectured outcome – Increased knowledge of the purchasers’ values

Table 26 shows that the majority respondents (19/24) stated that the approach had helped them (‘to

some extent’ or ‘a lot’) to understand what outcomes the purchasers of the project valued.

Table 26 Extent to which staff felt that the MSC process had affected their understanding of the

outcomes valued by the purchasers against meeting context

Meeting context *

Response 1

South West2

Northern Irrigation

3Gippsland and

North East

Totals



No effect 2 2 1 5



Totals 7 7 10 **n=24 Gamma value =0.48, approximate significance =0.06, No statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 4 non-respondents

Nevertheless, five informants (17%) felt that the approach had had no effect on developing a more

fully shared vision. There was no regional difference in response, and in this case, all staff from all

regions received the same quantity and quality of feedback. They all received the booklet Target 10

evaluation stories which contained the purchasers comments, in addition to receiving verbal

feedback presented at a meeting which all project staff attended.

The comments associated with lower scores tended to be concerned with the nature or extent of

feedback that was given from the round table meeting of purchasers. For example, one respondent

explained that as the purchasers were quite divided in their views of the stories, the feedback had

not been particularly useful:

The process still left us a little unsure of what areas the funders were keen to see stories on. As the

feedback from them was more confused (quote: extension staff).

Another respondent suggested that more follow-up was needed after the round table meeting:

226

Project funders response was a ‘one-off’ event and needed to be reinforced by some action to have

any significant impact (quote: extensions staff).

During discussions, several participants explained how they felt that the feedback from the

purchasers should have been more frequent; as it only occurred at the end of the year, there was no

possibility of adapting the approach to meet their needs in the short term.


This conjectured outcome (increased staff understanding of other people’s values) did not appear to

be strongly related to the conjectured context (that this outcome would occur where meetings were

conducive to open discussion and open exchange between staff and farmers).

Yet staff members felt that the MSC process had contributed to some extent to an increased

understanding of the outcomes valued by both the state executive committee and the purchasers. In

both cases, it appears that this outcome is more likely to be achieved where the feedback from the

story review process is circulated at regular intervals, where that feedback is constructive and where

it is clear in its message.

f) A more fully shared vision

________________________________________________________________________________

A further conjectured outcome was that the MSC process would enhance inter-organisational

reflection, communication and dialogue, and so result in a more fully shared vision. The

conjectured context-outcome relationship was tested by examining informant perceptions of

whether a more fully shared vision had been created in different meeting contexts. In this case the

conjectured mechanism was also examined. The idea was that the MSC process would lead to new

forms of inter-organisational communication and that by regularly challenging assumptions

(concerning the stories of change) this would ultimately lead to a more shared construction of what

is happening and what is desirable.


Mechanism Project stakeholders debate and reflect on project activities on a regular basis throughout the project. Inter-organisational reflection, communication and dialogue are enhanced.

OutcomeOver time, the increased communication and dialogue will bring about a more shared construction of achievements and what is desirable.

227

Conjectured mechanism – enhanced inter-organisational dialogue

During implementation of the MSC process, I observed that there was a marked increase in inter-

organisational communication; for example, feedback documents from the CEC meeting were being

regularly circulated amongst the committee members and extension staff. Increased dialogue was

also apparent at meetings at which stories were selected. In some regions, participants valued this

dialogue highly (illustrated by the quotations in Text box 9).

The opportunity to meet and discuss the stories helped to promote an understanding of each other’s viewpoint, if not necessarily a shared vision. Often in our work there is not enough time taken to reflect on what we have done and to learn from what we have done and I think the story approach is really good in that encourages us to do this (quote: extension staff).

The way people have been encouraged to talk and communicate during this process has been very beneficial for the project (quote: extension staff).

But I certainly think it is a good team building thing between extension officers and farmers or between the state exec and extension officers. But to have all the different heads all sitting around a story, deciding whether a story is good or not is very useful. But also it shows the different perceptions between all the different individuals and that is interesting and illuminating (quote: extension staff).

Text box 9 Comments associated with the benefits of discussing the stories

For some participants however, the most valuable aspect of the MSC process was the way that it

opened the door for dialogue with the purchasers (see Text box 10).

The approach was also really good in building up relationship with the purchasers - helped us to understand more about how they think – and that they do think differently and we need to report to them in different ways. That’s been a real eye opener. The stories really brought this point to a head (quote: project staff).

I think the stories process has opened up so many doors for us. It is the best thing we could have possibly done – just for the doors – the doors to the purchasers – and been a lead in for the rest of the evaluation. People have had a good experience with it and are more likely to be keen to be involved in evaluation in the future. Plus all the positive promotion it has done (quote: project staff).

Text box 10 Comments concerning the value of the MSC process for creating dialogue with

purchasers

228

Conjectured outcome – a more fully shared vision

To test the conjectured outcome-context relationship, staff were asked whether they felt that the

MSC process had helped create a more fully shared vision between all the project stakeholders.

Most respondents (20/25) stated that the MSC process had helped (to ‘some extent’ or ‘a lot’) to

bring about a more shared vision between all the project stakeholders (Table 27).

Table 27 Staff perception of the extent to which the MSC process affected the creation of a more

shared vision against meeting context

Meeting context *

Response 1

South West2

Northern Irrigation

3Gippsland and

North East

Totals



No effect 2 2 1 5



Totals 6 8 11 **n=25 Gamma value =0.37, approximate significance =0.17, No statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 3 non-respondents

There were no significant differences in responses between the different meeting contexts. From the

comments associated with this questionnaire item, it is apparent that respondents selecting the ‘no

effect’ option felt that while a shared vision has not yet been created, this was important and

possible in the long term. One senior member of staff explained how he saw the MSC process as a

dynamic vision capable of reacting to external change:

The story process is like a dynamic vision – the vision statement is static – this doesn’t help you to

respond to a new things – this mechanism is more ‘adaptive reactive’ – that’s what is different about

it. At the moment we are going through a period of price drop, deregulation and GST and we have all

of them coming within a year or so. And this is not taking into account seasonal things, or global

fluctuations in the market – so we do need a reactive mechanism to deal with this change together.

Even just giving people permission to come forward with negative stories – sometimes as an

229

organisation we are criticised for always looking at the positive outcomes – the good news. Perhaps

we should use the story approach to work more in the direction of learning (quote: project co-

ordinator).

Another informant stated that:

It is certainly important that people have a common understanding of what the project is about. It

[the MSC process] is like a living philosophy of what the project is about (quote: project co-

ordinator).


For this conjectured CMO, the data suggests that the mechanism is being fired in some contexts, but

that the time frame to achieve the outcome might be longer than the 12-month study. There

appeared to be no strong associations between the regional context and staff perceptions of

achievement of this outcome. Many participants felt that there were important increases in inter-

organisational communication and dialogue and that the MSC process had helped them to

understand the views of different stakeholders. Some respondents did feel that the approach

hadhelped to bring these views more in alignment and hence to create a more shared vision. Others

felt that this was likely to be a more of a long term outcome, but nevertheless an important outcome

to aim for.

230

g) Increased knowledge of how to run the process to meet the project stakeholders’ needs

________________________________________________________________________________

Another conjectured outcome was that the MSC process would encourage evolution and changes in

the MSC process itself – so that it could be adapted to better meet the project’s needs. The

mechanism behind this is that the regular feedback would provide new knowledge with regard to

the information requirements of different stakeholders. The iterative nature of the process means

that these requirements could then be built into the process with each successive round of feedback.

In this case the context-outcome relationship was not tested, but observational data were used to test

the extent to which the conjectured outcome was realised.

Conjectured outcome – evolution and change in the MSC process

Chapter 7 described ways in which the approach was modified to meet the information needs of the

project during the 12-month trial. These modifications included the provision of a system of

ensuring that data were collected in an ethical manner, developing a fifth domain to encourage

stories about ‘lessons learned’ and the nominating of ‘regional champions’ to co-ordinate the

process in each region. After the trial ended, extension staff and committees decided to lengthen the

time span between the review of stories at the Central Executive Committee meeting, to reduce the

time consumed in review of stories. All these changes provide evidence that the extension staff

have been constantly modifying the approach to meet their needs, and are likely to continue to do so

in the future.

I was initially surprised to realise that very few people I spoke to understood the aims of the process

in the way that I did, even though I had explained it them many times in the past. One of the classic

critiques of participatory action research is that often staff do not understand the process fully, or

that they construct the process differently. Nevertheless, multiple constructions of an evaluation

ContextThat the organisational structure can accommodate change. That stakeholders are willing to enter into dialogue with regard to information gained from the process.

Mechanism Feedback provides new knowledge with regards to the information requirements of different stakeholders. Communication, learning and dialogue are enhanced.

OutcomeEvolution and changes in the way the MSC process is run so that it can better meet the project’s needs.

231

process can also be seen as beneficial, especially if people construct different, yet beneficial

outcomes from the process. One of the project managers explained to me that the reason he valued

the MSC process most, was the very fact that it did offer different positive benefits to different

stakeholders:

In my mind it certainly has been a very successful process. Just the support of the Target 10 team

illustrates this – as we are all very busy – given half an excuse to offload something and we will. Yet

the commitment to continue is still there. I still don’t think I grasp the entirety of what the approach

can do. What I find interesting is that different people seem to get different things out of it. What I

find interesting and important is not necessarily interesting and important for other people – but the

important thing is that everyone that I have spoken to has got something out of it. Which goes back

to the street performance indicator of participation rates – normally there is only a very small number

of people interested in the numbers - but in this technique everyone gets something out of it. So there

is more commitment to it (quote: project staff).

The MSC process is complex and operates across a whole organisational hierarchy. One thing that

shone clearly through the data was that people see different aspects of the process, depending on

where they are located in the project hierarchy. This is not necessarily a negative point, if people are

getting something they need form the process at the hierarchical level from which they view the

process. Indeed, for any one evaluation approach there may exist several different, yet functional

constructions of how the approach works. The conjectured CMO configurations present just one

plausible construction of how the MSC process worked across the Target 10 Project. This highlights

the importance of sharing and modifying the model with several key informants.

However, the fact that there are multiple constructions of the MSC process reinforces the need to

have someone (or a small team) driving the whole process, and that these people should have an

understanding how the process works at a range of levels. During interviews, several people pointed

out the importance of having someone with good organisational and facilitation skills to drive the

whole process. They expressed concern about what would happen after I stopped fulfilling this role.

It appears that the MSC process did allow for modifications to the process itself, to better meet the

project’s needs. However, participants held different constructions of what this process was; this

appeared to be related to the hierarchical position of the participant in the project. I suggest that the

process therefore needs someone or a small team to drive it, and to understand the process at

multiple levels. For this conjectured CMO configuration the relationship between context and

outcome were not explored.

232

h) Beneficial operational and strategic changes in practice

________________________________________________________________________________

In the above conjectured context–outcome relationship, in conducive contexts beneficial operational

and strategic changes would occur. As in the other CMO configurations it was hypothesised that

this outcome would occur in meeting contexts conducive to open discussion and exchange of views

between staff and farmers. The postulated mechanism behind this outcome was that, at an

operational level, extension staff would try to reproduce the outcomes deemed successful in the

stories and avoid the events described in the stories of ‘lessons learned.’ At a more strategic level,

decision-makers would make changes to the project through what they have learned in terms of

evidence of impact, and stakeholder values of this impact.

Conjectured outcome – beneficial and strategic changes in practice

During interviews, key informants explained how they used the stories for various purposes in their

extension practice including to explain a point to farmers or other extension staff and to help plan

future activities. To quantify the extent to which stories were used, and to understand whether they

were more likely to be used in certain project contexts, the questionnaire included some questions

about use of stories. Table 28 shows that around 50% of the respondents stated that they had used

the stories for each of the named purposes.

Table 29 shows that the extent to which respondents used the stories was significantly related to the

regional context. Respondents from Gippsland and the North East were more likely to use the

stories for all three of the named uses than respondents from other regions. The comments

accompanying this question suggest that the lower use by respondents from the Northern Irrigation

Region reflects the fact that there are several new staff who have had little chance to use the stories.

There is no clear explanation of why more respondents from the South West reported using the

stories less than in other regions, but it may be associated with the generally less positive reaction of


Mechanism Staff try to reproduce outcomes deemed successful and avoid traps described in the stories of ‘lessons learned.’ Decision-makers change attitudes in light of knowledge gained. They may also examine specific stories to help in planning.

Outcome Extension staff make operational changes. Those involved in planning make strategic changes to the project through what they have learned in terms of evidence of impact, and stakeholder values of this impact. Purchasers make more informed funding decisions.

233

participants from this region. This may be associated with the low participation of farmers in the

South West, and large formal meetings that appear to be less conducive to discussion and debate.

Table 28 Extent to which staff used the stories

Category % of responses (n=29)

Used stories to help explain a point to farmers

54%

Used stories to help explain a point to other staff members

50%

Used stories to help plan a new activity 54%

Table 29 Extent to which staff used the stories against meeting context

Meeting context *

Response*** 1

South West2

Northern Irrigation

3Gippsland and

North East

Totals

Have not used the stories 2 5 1 8

Used the stories for one of the named uses

2 1 0 3

Used the stories for two of the named uses

3 1 4 8

Used the stories for all three of the named uses

0 1 5 6

Totals 7 8 10 **n=25 Gamma value =0.51, approximate significance =0.003. Statistically significant difference at p<0.01. * The meeting context was scored on a scale of 1-3, where 1 is low and 3 is a high. The scale refers to the degree to which the meetings (to which respondents attended) in different regions were conducive to open discussion and exchange of views between staff and farmers (see Table 18). ** There were 3 non-respondents *** The three ‘named’ uses for stories were (1) to help explain a point to farmers, (2) to help explain a point to other staff members (3) to help plan a new activity.

Stories as a tool to help extension officers to explain things to farmers

An outcome of the MSC process was that several extension officers (46%) recalled stories as part of

their normal extension practice: e.g., to explain certain points to a farmer. This is illustrated by the

following quote:

I frequently relate stories to help explain a point to farmers. Yes, I have used some of the stories that

have come from the approach – especially for a group of farmers who are wondering what they are

going to get out of the process – like they can learn to do the calculations themselves, and talk to

people sales reps more confidently - and so on (quote: extension staff).

234

One informant explained how extension officers utilise this approach naturally, by recalling to

farmers, stories derived from their own experience, but were now co-opting the stories from the

MSC process into this technique. In the long term, augmenting the folk wisdom of extension

officers may lead to a more unified understanding of possible project outcomes. This use of stories

may be particularly beneficial to extension officers with little field experience. However, the use of

stories to explain project outcomes and practices is not necessarily related to operational changes in

extension practice.

Stories as a communication tool with other staff and stakeholders

Several respondents (46%) explained how they used certain stories to communicate with other staff,

or with people from outside the project, especially in relation to the nature of their work:

Actually I had to give a seminar to the research staff, and I read one of the stories out. It illustrated

how complex extension is – that their research is just one part of the complexity of practice change.

The story helped them to understand the whole process of extension. They seemed quite surprised at

how complex it all was (quote: extension staff).

Stories as a tool to change extension practice

There were few examples of extension staff significantly changing their extension practice though

the learning gained from the stories. However, one respondent was adamant that this outcome was

occurring and relayed an example from his own practice:

I have an example! The story ‘turned a loss into a profit’, was seen very positively. This was about

a real ‘aha’ moment when a farmer came along and picked up the concept that the extension officer

had been explaining and then proceeded to explain it to all the other farmers. The farmer in the story

had taken the whole group with her. I followed the example in the story, encouraging the whole

group to work through the issues as a group, and they were beginning to identify the same issues as

the farmers in the story! (Quote: extension officer).

Stories as a tool for guiding planning

Fifty-three percent of staff reported that they had used the stories as part of a planning process.

Respondents explained that they had examined the stories and the interpretations as a part of

planning process of new activities. It appears that stories of lessons learned were seen as

particularly useful in this regard.

235

During interviews and discussions, it emerged that the stories were also being used in a number of

other ways, including providing information to include in the evaluation report.

Stories used to provide information for the evaluation report

Towards the end of the 12-month trial, extension staff were becoming increasingly involved in

writing a final evaluation report for the end of the first phase (five years) of funding for their

extension project. This report was to be sent to the project funders, and was a large effort, involving

many staff. This document was expected to be important in terms of influencing purchasers’

decisions on allocation of funds for a second phase of the project. The quotations in Text box 11

illustrate how the stories were especially valued when it came to writing the evaluation report.

Many of the stories or parts of the stories were included in the evaluation report, and placed next to

quantitative data that had been collected on the achievement of valued outcomes of the individual

programs. The stories were seen to add another dimension to the information.

I had to write the evaluation report for the DBF program – and a lot of evidence - especially at the higher levels of Bennett’s Hierarchy came from the stories. There would have been a real gap otherwise. So with the stories it is an easy way of documenting that higher order stuff. When I was writing the impact section of the evaluation report, it was great to have the stories. It would have been drier and less personal without the stories (quote: extension staff).

Stories are a valuable evaluation tool and help complete the picture. We can have used the stories with funding bodies, farmers and steering committees to highlight the outcomes and benefits of the T10 programs (quote: project co-ordinator).

I think it has helped us capture a whole lot of things that we would never have captured because it has been an on-going thing. So the value has been that it has put another dimension on to the evaluation. But I know there are other things that we have to do, like collecting names and stuff. But this collects the personal experience which adds another dimension – that I thought was really good (quote: extension staff).

Text box 11 Comments associated with the value of stories for evaluation reporting

236

Stories as a tool for guiding strategic changes and organisational restructure

There is no evidence that the stories strongly influenced the project redevelopment in terms of

large-scale strategic changes, which occurred shortly after the 12-month trial ended. Some

respondents felt that a system of responding to the lessons gained from the stories was lacking:

I don’t think that we have set up a system to respond to the lessons learned from the stories – so this

has not happened…The limitations are that it can take a fair amount of the time during meetings, we

perhaps did not as a group sit down and say, well here is all this great information we are generating,

how are we going to use it? (Quote: committee member).

Informants also inferred that the way in which the stories affect operational and strategic change is

less than direct, as strategic changes are obviously influenced by other factors in addition to

information gained from the MSC process. Indeed there is no evidence of any major strategic

changes being made that were a direct result of something learned from a story (see Text box 12).

Yet, there was a feeling that the process had influenced strategic planning at the project level.

There was no evidence to suggest whether or not the purchasers had used any of the information

gained from MSC process to make decisions about funding allocations. All that can be said is that

they were supportive of the process, and keen to be involved in the process in the coming years.

I would find it difficult to think of an individual story that has radically altered anything in Target 10 – possibly stories like ‘outdoing the consultant’ might trigger actions in people or encourage people. But I think that the stories themselves carry messages that need to be provided with a supportive environment – you can’t just say to someone ‘if you do this you will get this result’. You have to provide a whole environment – the stories are one of the things that help to encourage them. The stories provide peer group support for other farmers to try new things (quote: project co-ordinator).

It’s like in extension – when you make a decision, lots of things come together to influence you. The stories play a role in confirming what we already believed and this supports us in making the decision, along with other things. The stories play a role in helping you make the decisions. In some ways they also challenge things that you thought you believed. But I don’t see too many stories that really blew me out of the water. They either confirm what we already know and others challenge us a bit (quote: project co-ordinator).

Text box 12 Comments referring to how stories have affected change at a strategic level

Patterns between the conjectured context and outcome

There appeared to be some relationship between the different regional contexts and the achievement

of the conjectured outcome. The region in which staff and stakeholders were most actively involved

in the MSC process was also the region from which respondents most frequently reported using the

237

stories in planning and extension.

There was some evidence that the stories were influencing operational and strategic changes in

practice. Nevertheless, some participants expressed the view that the outcome of operational and

strategic changes in practice had not been fully achieved. The stories were used in a number of

ways that were not anticipated, especially the use of stories in the practice of extension. One further

use of the stories and feedback was to provide evidence for inclusion in the final evaluation report

submitted to the purchasers.

i) Organisation develops a sustainable capacity to learn and change

________________________________________________________________________________

A final conjectured outcome of the MSC process is that by having a process in place to examine and

challenge the value of what is occurring at the field level, the project team would gain an improved

and sustainable capacity to react to the changing external environment. This external environment

represents farmers’ as well purchasers’ requirements. It was postulated that this outcome would be

more likely to occur in contexts where story review occurs in meetings that are conducive to open

discussion and exchange of views between staff and farmers.

In the model building process, we envisaged that this conjectured outcome would take considerable

time to fully realise, as it requires long-term changes in organisational culture. It is perhaps the

ultimate goal of the MSC process. So the conjecture was not tested – as there was no expectation

that the outcome would have been realised in the period of 12 months. Yet the other outcomes

mentioned in the evaluation theory model could, in effect, be ‘stepping-stones’ to the achievement

of this final outcome. There is no evidence to date that this final outcome has been achieved but

several of the key informants believed that the MSC process does have the potential to realise this

outcome in the long term. Other outcomes that we hypothesised were stepping-stones to this final

outcome have been achieved. This is a positive indication that institutional learning and


Mechanism Project members learn how to better detect, reflect, and collectively respond to outcomes at the field level and to respond to changes in stakeholder concerns and wishes.

Outcome By having a process in place to examine and challenge the value of what is occurring in the field, the project team may gain a sustainable capacity to react to the changing external environment.

238

sustainability could be realised if the project were to continue with the MSC process.

8.5 Summary of Testing Model

The conjectured CMO configurations illustrate the theory that the MSC process brings about project

improvements through involvement and illumination of program stakeholders, dialogue, beneficial

changes in extension practices, and organisational learning. The data suggest that most of these

outcomes are occurring for specific groups of people. This indicates that the MSC model is a

plausible, ‘implementable’ model that can lead to most of the postulated outcomes in at least some

contexts.

However, there appear to be certain contexts more conducive to the achievement of these outcomes.

Positive outcomes appears to be associated with:

• high participation of farmers in the story selection and collection process

• small informal meetings where selection occurs

• a skilled facilitator available to run the review sessions

• regional champions with good organisational skills, and perhaps ability to motivate people to

narrate/write the stories

• voluntary rather than forced participation of people in collecting the stories

• sufficient contact between extension workers and clients for the stories to be generated in the

first place

• a space for story review, where the emphasis is placed on participation rather than competition

• extension staff attendance at some of the review sessions at the regional or state level to

understand how farmers react to the process

• regular and constructive feedback from the regional committees to the extension staff with

regard to which stories were selected and why

• situations where the story review process is not conducted by the same group of people who

have written the stories, especially when there are social factions present in the group.

239

8.6 Impact of the MSC Process on the Project

Extent to which process met the evaluation needs of the project managers and co-ordinators

At the start of the 12-month trial, the project managers articulated that the evaluation needs of the

project were to:

• gain an overarching understanding of the impact, (including unexpected outcomes) of the

project on farmers’ lives

• employ an evaluation process that was capable of fostering dialogue and negotiation between

the different project stakeholders (as discussed in Section 5.2)

• employ an evaluation process that would combine well with the existing portfolio of evaluation

activities.

With regard to the first aim, evidence suggests that the MSC process helped many stakeholders and

extension staff to gain a better understanding of project impact. A wide range of outcomes, many

beyond those expected, were captured by the stories (see Chapter 7, Table 10). With regard to the

second aim, the data have demonstrated that the MSC process was able to adequately foster

dialogue and negotiation between many different stakeholders. The third aim also appears to have

been met, in that the stories were used to complement and add a ‘human touch’ to the quantitative

findings in the final evaluation report that was submitted to the purchasers.

While project staff and managers stated that the MSC process was of as high importance to their

evaluation needs as the other evaluation initiatives, it was always regarded as an addition to the

important ‘goal-based evaluation’ of individual programs. It was a telling sign that in the final

evaluation report (sent to the project purchasers), the findings of the cost benefit analysis were

placed at the front, followed by the evaluation of the extent of achievement of valued outcomes for

individual programs. While all of these sections included some isolated stories to illustrate points,

the findings of the MSC process were placed in the appendix of that report.

240

The MSC model offered other positive outcomes for the management that they had not anticipated

before the trial began:

It has given us more information than we had before – it is have given us a better evaluation

outcome. It has bolstered our evaluation capacity, improving our credibility, improves our position

in securing resources. And helped in training staff in terms of working out what evaluation is all

about. I think we are more aware of what we are trying to do – it has really contributed to that

process (quote: project co-ordinator).

At the onset of the trail, a focus group was held between the four regional-coordinators of the

Target 10 Project, to capture their expectations of the process. The expectations tended to be

concerned with the formative aspects of the MSC process, for example:

It think it will bring to an action stage a lot of things that, I believe, would have been there in the

past, but we never actually captured them to the same extent, or acted upon them to the same extent,

as we have potential do here…I think that there will be a lot of things that get fixed up as a result of

the process… (quote: project co-ordinator).

Other expectations included that the approach would have beneficial spin-offs for marketing and for

getting farmers involved in evaluation. From the evidence presented in this chapter, it seems that

the approach has met many of these expectations. However, it may not have brought about as many

tangible changes to extension practice as anticipated by the regional co-ordinators.

Extent to which process met the evaluation needs of the extension staff and committee members

Prior to the 12-month trial of the MSC model, the evaluation requirements of 150 project

stakeholders were surveyed using a Delphi approach (see Section 6.3). This included all project

staff, all committee members and a number of key stakeholders of the project. A consensus was

reached with regard to the domains of change that were to be used to guide the story selection

process. Thus, from the onset, the collection of evaluative information was guided by the needs

articulated by the project stakeholders.

Qualitative data from this study suggests that most stakeholders felt that the outputs of the approach

had met their information needs. However, there was suggestion that information about

productivity and profitability had not been adequately captured. At times the purchasers suggested

that the stories would be better if they contained more background information.

241

Where the context was suitable, staff perceived positive outcomes that they had not anticipated

from the MSC process, i.e., that:

• the documentation of stories and memory of the stories augmented the collective memory of the

project

• the process was particularly enjoyed by the farmer committee members, and some perceived

that this helped the farmer committees to have a voice in the evaluation and to contribute to the

project

• the sharing of stories had a positive impact on staff morale

• the MSC process helped staff and stakeholders to understand project impact

• the MSC process helped staff to understand what outcomes were valued by different

stakeholders

• the process of story selection encouraged positive dialogue concerning the ‘living philosophy’

of the project

• involving the purchasers in the process was seen as an important step in opening doors to

negotiation

• the stories were used in a variety of ways in extension practice and in planning

• participants felt that, in the long term, the MSC process could bring about a more shared vision

between all the project stakeholders.

Extent to which the MSC process met the evaluation needs of the purchasers

The purchasers were not asked to specify their expectations of the process before the trial. In

hindsight this would have been a beneficial addition to the research. Nevertheless, as mentioned in

Section 3.5 several reports have explicitly stated the evaluation needs of the purchasers from the

Agriculture Division of NRE. The extent to which these needs were met will be discussed in

Chapter 9, as well as an examination of the broader values of the MSC model.

However, at the round-table meeting of purchasers, and during the focus group that followed it, the

purchaser group appeared to find the approach highly valuable. In general, they suggested that the

approach was valuable in that:

• the stories demonstrate more than just technical outcomes

• the stories demonstrate many unexpected outcomes

242

• the review process of the stories demonstrated the organisational capacity of the project

• the stories captured the complex nature of farming system. However, one purchaser suggested that the stories needed more background information to fulfil his

requirement for evaluation information:

I think it’s pretty unclear to the uninitiated what the changes actually mean in tangible terms. It’s

hard to know what 1,000 kg of solids per ha means if you’re not a dairy farmer. A lot of people who

are interested in these stories are not dairy farmers. I think the story is good, but a bit is missing. A

lot of the stories don’t go far enough, they don’t take you right through to what it really means

(quote: purchaser).

The purchasers suggested that the quantifiable figures in these stories should be converted, where

possible, into dollar values. This would help people from other industries to understand the

significance of the changes described. They also suggested that specific types of stories could be

targeted towards different evaluation audiences. For example, those stories regarding profitability

should spell out the tangible outcomes in dollar figures and be accompanied by background data,

such as regional production benchmarks.

Nevertheless, the overall sentiment was that the MSC process was valuable, and all eight

participants (the purchasers) elected to participate in a similar process in the following year. The

participants seemed particularly pleased with the holistic nature of the information that the stories

presented: What struck me, was just how valuable it is, to be actually able to capture the much wider

ramifications of what we are doing rather than the very narrow one that we tend to walk down

(quote: purchaser).

Unexpected negative outcomes of the process

There were instances of non-constructive, competitive pressure between staff with regard to having

stories selected, over which that they felt some ownership. In some instances (and regions) staff

also felt that the MSC process was just another thing to do on top of an already busy schedule. Of

all the limitations and issues mentioned in the interviews and discussions, the ‘time factor’ was

mentioned most frequently. It could be that the amount of time consumed by the approach had

negative effects upon other work.

243

8.7 Conclusions

The outcomes stipulated in the conjectured CMO configurations (constructed on the basis of the

collective experience of eight key informants and myself) did appear to be realised in at least some

of the project contexts. The model appears to represent a valid construction of how the MSC

process can bring about organisational learning and provide evaluative information. Using Pawson

and Tilley’s (1997) Realistic Evaluation model to guide the meta-evaluation, the concept of

mechanisms being ‘fired’ in certain contexts to bring about outcomes was found to be valuable. It

helped draw out how outcomes were achieved, and the particular contexts in which this occurred.

The intention is that information gained in this case study will provide some general guidelines for

the sort of extension projects that might and might not gain value from the MSC process. Chapter 9

will examine the extent to which the MSC model represents a good, valid approach to evaluation,

and the extent to which it could usefully add to the ‘basket of choices’ for evaluation of extension in

Australia.

244

CHAPTER 9

THE MOST SIGNIFICANT CHANGE MODEL FOR

EVALUATION IN AUSTRALIAN AGRICULTURAL

EXTENSION

In this chapter, the second part of the meta-evaluation of the MSC model is presented. Here,

the focus is on the merit of the approach in general, and the extent to which the MSC model

is appropriate to agricultural extension projects in Australia. To do this, the MSC model was

examined using the key meta-evaluation questions that were developed in Chapter 2. I

concluded that the MSC process could contribute usefully towards meeting evaluation needs

at the national level. In terms of the MSC process contributing to project improvements, the

capacity of the MSC model will depend on the individual context of the project in which it is

applied. I suggest that this approach is particularly appropriate for large extension projects

with diversity in outcomes, with a high degree of farmer participation in project planning and

decision-making. While the model satisfies some of the premises for good evaluation, it also

has inherent biases and weaknesses. This emphasised the need to combine the MSC model

with complementary evaluation approaches, to help meet all evaluation needs and also to

offset the bias inherent in the approach.

9.1 Introduction

In Chapter 8, the first part of the findings of a meta-evaluation of the Most Significant Change

(MSC) model were presented. The meta-evaluation focused on the extent to which the MSC model

was able to meet the needs of the case project stakeholders and to contribute to project

improvements. As the aim of the thesis is also to develop a model of evaluation for agricultural

extension projects more broadly, it is important to evaluate the MSC model at a more macro level

245

too. The aim of this chapter, therefore, is to determine the extent to which the MSC model meets the

premises of ‘good’ evaluation. This includes the extent to which the model is appropriate for the

project context – in this chapter that context is extended to extension projects across Australia.

Lessons learned from the case study evaluation are examined to provide clues as to the applicability

of MSC model to extension projects more broadly. The Target 10 Project cannot be considered

representative of all large extension projects in Australia, and it is unlikely that other projects would

have the same outcomes and reactions to the MSC process, if implemented. Nevertheless, the

understanding gained in terms of the CMO configurations (presented in Chapter 8), will provide

some indication about the sort of project contexts in which the MSC model is likely to bring about

desirable outcomes.

The chapter begins by recalling the meta-evaluation questions that were developed in Chapter 2,

and applying these to the MSC model. Following this, I discuss the extent to which the MSC

process is able to offer a useful addition to the ‘basket of choices’ for extension evaluation in

Australia. Recommendations are made about the sort of extension projects that might gain more

fully from the MSC model to evaluation, and what evaluation approaches are likely to be

complementary.

9.2 Applying the Meta-Evaluation Questions

In Chapter 2, eight meta-evaluation questions were developed, in order to help determine the

‘goodness’ of an evaluation approach. The questions were:

• Is the evaluation relevant to the context and purpose in hand, and does it fulfil the evaluation


• To what extent can the evaluation contribute to the project being better able to meet needs?

• Is the evaluation guided by program theory?

• Does the evaluation make an attempt to account for different program outcomes?

• Are the evaluation processes socially and politically just?

• Is there an appropriate ratio of costs to benefits for the project from the process (and outcomes)

of the evaluation?

• To what extent are the design and data collection/ analysis of the evaluation valid?

• Does the evaluation attempt to judge the merit and worth of projects?

246

In the following sections (9.2 to 9.9) the MSC model is considered in terms of its ability to meet the

needs of evaluation in Australian extension programs, in terms of each of the above questions.

Lessons from the case evaluation (with the Target 10 Project) are also drawn out to provide

indications of the applicability of the MSC model to extension projects more broadly.

9.3 Is the Evaluation Relevant to the Context and

Purpose in Hand?

In Section 8.7, I claimed that the MSC model was able to go a considerable way in meeting the

articulated needs of the Target 10 Project management, the staff and the purchasers of the project.

This was ultimately verified by their commitment to continue with the MSC process after the trial

was complete. Nevertheless, while project staff and managers stated that the MSC process met their

expectations, the model was always seen as an addition to the important ‘goal-based evaluation’ of

individual programs.

Extent to which the MSC model can address the challenges posed by changes in extension

In Chapter 3, recent changes in the philosophy that underpins agricultural extension and the

organisations that deliver extension were examined. I argued that these changes have influenced

the way publicly-funded extension projects are planned, managed and valued. I concluded that the

changes represent unresolved dilemmas for evaluating the new genre of projects, therefore there is a

need for evaluation that can:

• provide evidence on the extent to which predetermined objectives have been achieved


specific outcomes

• provide a process whereby farmers can participate in evaluation

• be conducted as part of an internal evaluation capacity and hence foster organisational learning

• provide performance information for communication between multiple stakeholders

• involve processes that foster dialogue between multiple stakeholders.

247

Provide evidence of the extent to which predetermined objectives have been

achieved

Recent changes in public sector management have seen an increased demand for the summative

evaluation of projects, to provide accountability to both the ‘purchasers’ and the farmer-clients. In

the case evaluation, the MSC process was seen to help stakeholders understand impact, yet the

emphasis was squarely on ‘understanding’ rather than ‘proving’. The MSC process was not

designed to provide the sort of data that is generally accepted as ‘highly objective’ (e.g., for

summative evaluation purposes). Nor did it provide quantitative evidence with regard to the extent

of achievement of predetermined objectives. In the case evaluation this need was fulfilled through

other evaluation techniques.

It could be argued that in using the MSC model for accountability it would be all too easy to hide

negative outcomes. The MSC model privileges the voice of those inside (or close to) the project –

those most likely to be advocates of the project. It does not seek the views of antagonists of project

practices. In addition, the MSC process does not give an indication of the spread of adoption of

technologies across the dairy farming population. Instead, the MSC model is designed to capture

‘remarkable events’ rather than the average experience of farmers. The MSC model is an example

of formative evaluation, aimed at encouraging learning between project stakeholders and

‘improving’ the project, rather than ‘proving’ what the project has achieved. Consequently, if used

in isolation, the MSC is unlikely to satisfy stakeholders’ demands for summative evaluation

concerning the achievement of predetermined objectives.

Provide accountability for group-based models of extension that produce

diverse, context-specific outcomes

In instances where projects have few or no pre-determined outcomes (but instead are more process-

orientated) and produce diverse and context-specific outcomes, the MSC model may be able to play

an important stepping-stone to summative evaluation. Davies (1996) suggests that the MSC process

may be best viewed as a ‘search mechanism’ for unexpected or unanticipated outcomes. Where

outcomes are not predetermined the function of ‘searching’ for outcomes could be very important.

However, to provide summative information, the MSC model would still need to be coupled with

some more empirical research and analysis. In the case evaluation, the stories were analysed and

grouped into themes with the help of Bennett’s Hierarchy. A further step could have been to

conduct a survey across a representative sample of participating farmers to quantify the spread of

248

these ‘emergent’, valued outcomes. The idea is that the MSC model would be used to search for

unanticipated, valued outcomes, then coupled with a system of measuring these ‘dynamic

indicators’ (replacing the need to measure predetermined, ‘static’ indicators over the entire life of

the project – as is practised in conventional monitoring). Davies (1996) suggested that the MSC

model could accommodate a stage of quantification in this manner, but this was not implemented in

either the Bangladesh case, or the Target 10 case. In the Target 10 case, this was not seen as

necessary, as there were other evaluation process being conducted to fulfil this requirement.

Provide a process whereby farmers can participate in evaluation

The MSC model could be particularly relevant for evaluating the new genre of participatory, group-

based projects. It can accommodate a wide range of participants in the evaluation process itself, and

there is no reason why this could not accommodate more farmers than were involved in the case

evaluation. As the structure of the story review process ‘rides on the back’ of a pre-existing project

structure, if farmer-clients are more involved in that structure, they would also be more involved in

the story review process. To promote the participation of farmers in the process, the first stage of

story collection and analysis could occur during farmer meetings. In Section 3.3, I advocated that,

in highly participatory projects, the process of evaluation itself should reinforce, rather than inhibit,

the effort towards participation and empowerment of participants. I argued that highly participatory

projects require participatory evaluation; approaches that allow the stakeholders and beneficiaries to

have a say in which changes are important, and which changes should be measured. The MSC

process has potential to meet these requirements. In the case evaluation the MSC process was

effective in helping staff and farmers to understand critical incidents surrounding success and

failure.

Evaluation that can be conducted as part of an internal evaluation capacity

and hence foster organisational learning

Evidence from the case evaluation suggests that the MSC model is ideally suited to provide internal

evaluation to foster organisational learning. Firstly, the MSC model is internally driven at the

project level, which is in line with the trend to promote internal evaluation with facilitation by

evaluation experts. Secondly, the MSC model seeks to enhance the organisational learning capacity

of the project, which is one of the demands being made by the ‘purchasers’ in the Victorian

Agriculture Division. In Chapter 8, evidence was presented to suggest that the MSC process

contributed to the achievement of several outcomes in the case project that could support

249

organisational learning, i.e., the outcomes were:

• enhanced organisation memory – an aspect of organisation learning that is seen to be especially

valid under times of rapid change

• encouraged staff to reflect on the impact of their extension practice

• helped staff and stakeholders to understand project impact

• helped staff to understand what outcomes that were valued by different stakeholders

• encouraged positive dialogue concerning the ‘living philosophy’ of the project

• resulted in stories being used in a variety of ways in extension practice and in planning

• had potential in the long term to bring about a more shared vision between all the project

stakeholders.

Provide performance information for communication between multiple

stakeholders

The stories and interpretations together represent the performance information produced by the

MSC model. The stories contain a wealth of information, both factual and tacit. The Target 10

implementation of the MSC model revealed that the stories were perceived to be ‘powerful and

engaging’ for those who read them. The interpretations themselves represent another domain of

information, and reveal what stakeholders do and do not value in terms of impacts of project

activities. I suggest that the MSC model can produce highly appropriate performance information

for the purposes of communication of project impacts between multiple stakeholders.

Involve processes that foster dialogue between multiple stakeholders

Evidence from the case evaluation suggests that the MSC process is well designed to foster

dialogue between multiple stakeholders. The MSC model is particularly suited to deal with

numerous organisations working in partnership to bring about extension outcomes as was seen in

the case study where a whole range of stakeholders were drawn into dialogue with one another. The

on-going nature of the MSC model supports an iterative process that can accommodate emerging

information needs of a range of stakeholders.

Like many large extension projects, the organisational structure under which the Target 10 Project

operates is complex and several purchasers and co-providers interact in a dynamic project

environment (McDonald and Kefford, 1998). Taking this organisational complexity into account,

250

it is vitally important that time is allocated for ensuring that the various stakeholders enter into a

meaningful dialogue about happenings in the field, and whether these experiences represent

desirable outcomes. At the round table meeting (Section 7.3), it became apparent that different

purchasers interpreted the stories in differing ways and held different things to be of value. This

finding supports the concept that evaluation is conducted in a value-pluralistic context (Toulemonde

et al., 1998). Thus the MSC process encourages negotiation and dialogue between the various

evaluation stakeholders (including the purchaser and the provider) which addresses at least some of

the challenges posed by changes in public sector management.

Summary

In general, the MSC process appears to go some way in addressing the needs of greater

communication of evaluation findings, managing organisational complexity and value pluralism,

but it does not provide the sort of data that is being demanded under the call for greater

accountability to clients and purchasers.

9.4 To What Extent Can the MSC Evaluation Contribute to

Projects Better Able to Meet Needs?

In the case study evaluation the MSC process was seen to lead to several outcomes that had

potential to contribute to project improvements. Using Pawson and Tilley’s (1997) Realist

Evaluation model to guide the meta-evaluation, an attempt was made to determine the project

contexts in which conjectured ‘mechanisms’ (inherent in the MSC process) were triggered to bring

about beneficial outcomes. I found that there were marked differences between the four project

regions across which the MSC process was implemented, and these were examined to try to explain

the different outcomes produced. Even within one project environment, certain contexts were more

conducive than others to the realisation of useful outcomes.

Following this logic, it can be assumed that some projects, are likely to benefit more than others

from the MSC process in terms of project improvement. The study revealed that beneficial

outcomes in terms of project improvements from using the MSC model are more likely to be gained

in projects with the following characteristics:

• high participation of farmers in the project development and in the committee structure

251

• small informal meetings where the story selection processes can occur, which emphasise

participation rather than competition

• skilled facilitators available to run to review sessions and to motivate people to narrate/write the

stories

• having the story review process conducted by a different group of people to those who have

written the stories, especially when there are social factions present in the group

• voluntary, rather than forced participation of people in the collection of stories

• close contact between extension workers and farmers for stories to be generated (if extension

officers are to narrate stories).

In addition to the above contexts, I suggest that the project structure itself is important. The MSC

model was designed to evaluate large projects, with several hierarchical ‘layers’ and a number of

project stakeholders. In the case of the Target 10 Project, the pre-existing structure appeared to be

well suited to MSC model. Nonetheless, there is no evidence to suggest that smaller projects would

not benefit from collecting, sharing and evaluating stories of significant change. However,

implementation of the whole model might not be possible or desirable in all projects.

9.5 Is the Evaluation Model Guided by Program Theory?

In Chapter 2, I suggested that good evaluation should consider the theories that underpin the

project. Considering the arguments of Pawson and Tilley (1997) and Chen (1990), good evaluation

should be guided by program theory, and should not be simply a ‘black box’ venture. Even Scriven

(1994), who levelled early criticisms at a theory-driven evaluation, recently conceded that program

theory is usually helpful, even if not always essential.

In the case evaluation, the MSC process was not guided by questions developed from a carefully

constructed model of the theory that underpins the Target 10 Project. Yet it did incorporate

questions guided by stakeholder concerns within defined domains of change. The MSC model

involves collecting impact information in an almost ‘goal-free’ manner and then coming an

understanding about key areas of findings through a process of induction. In the case evaluation, the

domains of change for the story collection (approximate to key evaluation questions) were

developed with considerable consultation with the management and stakeholders using the Delphi

process (see Section 7.3). In a sense, a trade-off was made between participation and ownership of

the evaluation process at the expense of a strong theory-driven development of key evaluation

252

questions. Perhaps this is quite a common dilemma in the evaluation of participatory extension

programs.

Program theory that is largely deductive in orientation (as described by Pawson and Tilley, 1997

and Chen, 1990), may be best suited to expert-driven evaluation rather than participatory

evaluation. Leviton (1994) argues that evaluation theory is most seriously deficient in assignment of

value to community-based programs. She suggests that when ‘experts’ frame questions for the

community they encounter understandable fury. The Target 10 Project is not exactly ‘community

based’ but it was influenced by theories of participation and community ownership. To some extent

the same issues as Leviton describes in community-based programs come into play. The MSC

model endorses the position that the clients’ and key stakeholders’ values are the ones that should

be used to assign value. Thus, use of a deductive-normative theory for developing the key

evaluation questions would be rejected in favour of developing key questions through the

achievement of some sort of consensus from the project stakeholders with regard to what they

believe should be measured in the evaluation. As a result of the lack of emphasis on drawing the

key questions from an explicit program theory, there is little analysis of the underlying mechanisms

that are triggered by the project.

However, there are other ways of incorporating program theory into evaluation that are more

inductive and allow for the inclusion of different stakeholder values. Patton (1997:224) describes

three varieties of program theory:

• User focused or ‘theories of action’ – were facilitators work with beneficiaries of a program to

articulate and specify their implicit theory of action. This model is largely derived from the

work of Argyris and Schon (1996) who contend that people in programs operate of the basis of

theories of action of which they may not be aware. Argyris and Schön (1996) distinguish and

try to draw out the contrast between two kinds of theories that people hold: ‘espoused’ theories

and ‘theories-in-use’.

• Inductive approach – involves doing fieldwork on a program to generate grounded theory. The

product of this is an empirically derived theoretical model of the relationship between program

activities and outcomes, framed in terms of important contextual factors. In inductive

approaches, the evaluator builds the theory from observations and fieldwork rather than from

discussion with those involved.

• Deductive approach – involves drawing on scholarly theories from academic texts and current

253

knowledge and testing them against empirical evidence. For the large part Pawson and Tilley

(1997) advocate a deductive approach; while they consider ‘folk-knowledge’ in their

conjectured program theories, these are not grounded in field work.

In the case evaluation, the MSC process did not incorporate any of these approaches to program

theory, but it could be modified to incorporate Patton’s ‘inductive’ or ‘user-focused approach’. I

suggest that there are two ways in which the MSC model could be usefully modified to incorporate

some program theory.

Firstly, the stories could be analysed en masse by a researcher to determine critical success and

failure incidents (as perceived by project stakeholders) and from this draw out a conjectured (yet

partially grounded) program theory model. This theory could then be tested against empirical data

to develop further knowledge of how the project works, and in what situations, and why. This

proposal, is similar to Patton’s inductive approach to program theory.

Secondly, the stories could be analysed en masse by a group of project stakeholders – and this

analysis form the basis of a conjectured (yet partially grounded) theory model of how the program

works. This process may need to be facilitated by someone versed in constructing program theory

models, but conducted with participation of stakeholders. The model itself could then be tested, as

is conducted in Realistic Evaluation by the same group. This approach lies somewhere between

Pattons’ ‘inductive-approach’ and his ‘user-focused approach’: it is inductive, as the models would

be grounded by data from the stories, yet unlike Patton’s inductive approach, it would involve

considerable participation of stakeholders. In contrast to Patton’s ‘user-focused’ approach, the aim

would not be to compare what extension staff say they do (espoused theory) with what they actually

practice (theory-in-use). Instead the intention would be for a group of project stakeholders to work

with the observations and constructions of program impacts (in the form of the stories and

interpretations) to develop a more sophisticated (and shared) construction of how the project works,

in what contexts, and why.

254

9.6 Does the Evaluation Model Make an Attempt to

Account for Different Program Outcomes?

The MSC model emphasises collecting as wide a range of outcomes as possible (in the forms of

stories), and in this sense is almost ‘goal-free’ in approach. Nevertheless, the use of ‘domains of

change’ does ensure that certain types of information are included in the evaluation. In the Target

10 case the domains of change were:

1. Changes in farmer decision-making skills

2. Changes in on-farm practice

3. Changes in profitability or productivity

4. Stories about ‘lessons learned’

5. Any other changes

These domains are left deliberately ‘loose’ so that those changes that are perceived to be important

by the storytellers can be captured through the process.

While the process may capture a range of unexpected outcomes perceived to be important by the

project stakeholders, it does not capture unexpected or unintended outcomes unless they are deemed

‘significant’ by those involved in the process (staff and committee members and purchasers). For

example, in Section 7.1, I mentioned that there has been some criticism of the Target 10 Project for

being focused too strongly on production, at the expense of social and environmental impact.

While there were multiple stories concerning negative and positive social impact of the program,

there was only one story concerning environmental impact of project activities. If stories had been

collected from known antagonists of project practices, then these sorts of views might have been

drawn into the process. It seems that, in the Target 10 case, the MSC process only picked up

unanticipated or unintended outcomes that were located within the discourse of those involved in

the project management (the staff and committee members).

This appears to be a serious omission to the evaluation process. However, I am of the opinion that

the Target 10 staff would have been willing to engage in discussion of stories from project

‘antagonists’ and that that this could easily be accommodated into the MSC process. Indeed, on

255

several occasions, staff mentioned the need for more stories about ‘lessons learned’ – because of the

high level of discussion and learning that stories about negative impact tended to generate.

9.7 Are the Evaluation Processes Socially and Politically

Just?

In terms of who was represented in the evaluation, it can be argued that the evaluation process was

not socially / politically just, in that it favoured the inclusion of some stakeholders over others.

Indeed, the story selection process is extremely biased in favour of those people who attend the

story review sessions. In the case of the Target 10 Project, the committees were usually attended by

an even spread of farmer members, project collaborators and staff. However, this was not the case

in one region, where farmers were greatly outweighed by the other members.

In addition, it should be pointed out that the farmers attending the project committees cannot be

considered fully representative of the wider farmer voice. At an early stage of the trial evaluation, a

wide range of people were asked about the extent to which the farmers on the committees were

considered ‘representative’ of the wider farmer voice (Dart, 1999b). There was general consensus

from informants (including farmers, industry representatives, staff from the University and staff

from the project) that the farmer members of the committees were not representative of the general

farming community (See Text box 13). However, this was seen as an unavoidable situation, and

informants felt that these farmers were able to articulate the wider farming community ‘felt needs’

to a reasonable extent. The wider farming community needs are also brought to the committee

forum by the industry representatives, who have a wider client base than the project staff (and

perhaps even some farmers).

256

They are as representative as can be expected. The other types of farmers wouldn’t attend the meetings. Hard-nosed, business-orientated, or less-articulate farmers would never go, but the ones that do go, do a reasonable job of representing the farming community. It is not perfect, but that is the best that you can do (quote: Industry representative).

They’re not representative. I don’t think they are at all. They’re the ones with the organisational skills, the interest and the curiosity and the time. Aren’t they? And I think, in a lot of ways, NRE falls into the trap of going back to the same people time and time again. They’re getting a select view. But then how do you get “Farmer Jake”, who doesn’t move from his farm, and, you know, knows nothing, goes out and plays sport? He would hate to sit in a room for a day. You don’t get them, do you? (Quote: Farmer representative).

As far as I am aware, there is no formal mechanism for the farmers [committee members] to gather the opinions of the farming community. As far I am aware, this is an informal gathering. I mean, they are out there farming with other farmers, and I presume that they get an idea about what other farmers want just from casual conversation and contacts. But I think most of the people on the committee are aware of what’s happening, and they are involved with discussion groups and those sorts of things, so they do provide feedback in terms of what those other farmers are thinking (quote: extension staff).

Text box 13 Informant views of the extent to which the farmer committee members ‘represent’

the wider dairy farming community

The MSC process is clearly embedded within the discourse of the project staff and committee

members. The farmer-clients who were involved in the selection process cannot be considered to

fully represent the farmer voice on matters of project practices and outcomes. Farmers on the

committees are generally advocates of the project, and thus there is a lack of the ‘critical voice’ in

the story review process. The MSC model makes no attempt to capture the opinion of those farmers

who choose not to participate in the Target 10 Project. So antagonists of the project may not have

been given a say in this evaluation. In some other evaluation approaches, processes are employed to

ensure that ‘antagonists’ are represented in the evaluation. For example, in Fourth Generation

Evaluation it is proposed that the ‘victims’ of a program are identified and their view sought (Guba

and Lincoln 1989). Inclusion of a similar process could add to the validity of the MSC model.

If the MSC model was combined with an approach to evaluation that did seek out non-participants

of the Target 10 Project and antagonists of the project, this could offset the bias and provide a more

comprehensive evaluation outcome. Another possibility is that a researcher could seek out stories

from antagonists and include them in the review forum, with stories containing elements of negative

outcomes (referred to as stories about ‘lessons learned’).

257

9.8 Is there an appropriate ratio of Costs and Benefits for

the Project from the Process (and Outcomes) of the

Evaluation?

Respondents reported that the biggest cost associated with the MSC process was the amount of time

that it consumed. While they felt that it was valuable to continue the process, they suggested that

the story review process should be less frequent. Since the trial, this change has been made; the

stories are being selected at every other committee meeting (every four months instead of two

months).

In an attempt to address this question, informants were asked whether they thought that the benefits

of the approach outweighed the costs in terms of time and resources, ie ‘was it worth doing at all?’

All informants felt that it was a worthwhile venture. They tended to see the main benefits as being

in the form of staff motivation, reflective practice in general and a new and very important

negotiation process with the purchasers. However, many respondents did not conceptualise the

process as a form of evaluation at all. Evidence of support for the approach is provided in the fact

that staff and committees voted to continue the approach, albeit in a modified form, after the 12-

month trial was complete.

It is also worth considering alternative approaches to the MSC model, and what the cost would have

been to the project if it had not been implemented. The question of whether another approach would

have produced the same ends more efficiently remains largely unanswered. Other qualitative

techniques, such as focus groups and in-depth interviews, while good at eliciting a wide range of

unexpected outcomes, are not continuous and would not capture the iterative feedback and learning

that is seen with the MSC model.

9.9 To what Extent are the Design and Data Collection/

Analysis of the Evaluation Valid?

At several occasions in the implementation of the MSC model across the Target 10 Project, people

expressed doubts about the ‘scientific validity’ of the output of the process. Some people were

258

especially concerned with the sampling technique and bias of the approach toward positive

outcomes.

Sampling technique

The sample technique of the MSC model is selective rather than inclusive, and will not provide

information on the ‘average condition’ of a farmer who attends the programs, but rather will

provide information about the exceptional circumstances, particularly successful circumstances.

This is referred to as purposive sampling. Some would argue that this is not an appropriate sample

technique to produce reliable information to enable judgements to be made about the performance

of the program.

Nevertheless, in qualitative research, purposive sampling (or purposeful sampling) is seen as a

legitimate form of data inquiry, forming a dominant part of the logic of qualitative research. Patton

states that:

The logic and power of purposeful sampling lies in selecting information-rich cases for study in depth.

Information-rich cases are those from which one can learn a great deal about issues of central

importance to the purpose of the research, thus the term purposeful sampling (Patton, 1990:169).

Patton goes on to describe several different strategies of purposive sampling, the logic of each

strategy serving a particular evaluation purpose. One category is ‘extreme or deviant case sampling’

– this approach focuses on cases that are rich in information because they are unusual or special in

some way. In the MSC model, the sampling system corresponds to this category where the logic is

to capture significant instances of success or failure. The purpose of this strategy is to learn from

these extreme stories, and ultimately to change extension practice to move towards more success

and away from failure. Therefore strategy is to select stories from which most can be learned.

If the purpose of the evaluation is to precisely document the natural variation among outcomes for

farmers who attend programs, then a random sample of sufficient size to be representative would be

needed to permit generalisations about the experience of all farmers attending programs. Whoever,

Patton (1990:170) suggests that ‘in many instances more can be learned from intensively studying

extreme or unusual cases than can be learned from statistical depictions of what the average case in

like’. Another option frequently practiced is to combine approaches, so that an understanding can be

gained of the normal distribution of participants, and the extreme cases. In the Target 10 Project, the

259

MSC process was combined with other approaches that did capture the normal distribution of

farmers attending programs.

Positive bias (and possible modifications)

In the case evaluation, the MSC model was biased in favour of success rather than ‘bad news’

stories. About 90% of the stories collected concerned positive outcomes. However, toward the end

of the trial, the staff and committees elected to include a further ‘domain’ named ‘lessons learned’,

to ensure that each region will present at least one ‘bad news’ story in each selection period. As was

previously described, the MSC model adopts a purposive sampling system, and captures instances

of success rather than the average farmer experience. However, in the case evaluation, several staff

pointed out that there seemed to be more potential for learning from the ‘bad news’ stories, and for

this reason, there was a move to encourage the collection of these stories.

The selection process is extremely biased towards the views of the committee members, and those

involved in the story selection process. However, unlike other research approaches, the bias itself is

offered up as another source of data about the organisational capacity of the project. The reasoning

behind the selection of certain stories was recorded and documented along with the stories

themselves. The inclusion of these interpretations as another form of evaluative data affords a high

level of transparency.

Reliability of factual content of stories

Early in the trial participants were concerned with the authenticity of the stories; ie., ‘how do we

know that they are not fictitious?’ All stories were accompanied by the names of those involved in

the event, and the location of the event. This meant that it would have been possible to follow up

the stories. Although such checking never occurred, there was considerable pressure by peers to

record information accurately; stories that seemed implausible, or incorrect in factual content, were

not selected.

Nevertheless, as was discussed in Chapter 5, the validity of the stories and interpretations is

somewhat dependent of the epistemological stance adopted. Under a constructivist epistemology the

stories represent reality in their own right, and it is in the telling of the stories that the storytellers

and reviewers construct reality. In addition to this, the interpretations that the tellers and the

reviewers provide give strong clues as to how these actors construct reality. The sharing of the

260

stories and of the interpretations made of these stories allows a new, shared and more sophisticated

reality, to be jointly constructed by the storytellers, reviewers and the story readers. In this sense

then, the process helps stakeholders apprehend reality (in its variously constructed forms) to make

decisions. Yet under an objectivist epistemology, the stories themselves require cross-checking and

may not represent the best way of accurately determining the events that have occurred.

9.10 Does the Evaluation Attempt to Judge the Merit and

Worth of Projects?

Scriven (1991) contends that evaluation should involve the determination of merit and worth of the

object being evaluated. In the case evaluation, the thing being evaluated was the Target 10 Project.

While stories of individual farmer outcomes were judged for their merit and worth by stakeholders,

there was no point at which the evaluators (stakeholders in this case) made an overall determination

of the merit and worth of the Target 10 Project. However, the stories and interpretations did

contribute to the evidence (in addition to the other evaluation activities) that was intended to help

the purchasers determine whether the project was of sufficient merit and worth to receive continued

funding and support. But this is not the same thing as the evaluators making a determination of

merit and worth of the project – as purchasers represent only one stakeholder group.

A similar criticism can be leveled at Guba and Lincoln’s (1991) ‘Fourth Generation Evaluation’

(FGE) model. Yet the MSC model goes one step further than the FGE model, which feeds interim

results back to the participants, but does not have a system to enable the participants to interpret and

judge the findings. In the MSC process during the story selection, participants discuss the relative

merit and worth of the individual outcomes (stories) and make judgements as to which stories best

reflect a significant account of change. Perhaps a further stage in the MSC process would be to

(periodically) analyse the stories en masse, rather than confining judgement to instances of localised

outcomes.

Level of extension outcomes addressed

Bennett’s Hierarchy can be used to help tease out the level of extension outcome that an evaluation

addresses, and whether the evidence is sufficient to make a legitimate judgement of worth and merit

about the program. Bennett (1977) lists seven levels of goals in extension (Section 4.5, Table 2),

and claims that it becomes more difficult to evaluate at higher levels of the hierarchy, as it becomes

261

more difficult to show that changes at these levels are the result of extension activity and not of

other factors. Conversely, changes at the lower level of the hierarchy are more easily attributed to

the intervention, but provide lower quality of evidence in terms of making valid determinations of

merit and worth. Using Bennett’s Hierarchy, good extension evaluation should include some

reference to changes at:

• ‘Level five’; changes in farmer knowledge, attitudes and skills, in addition to lower level

outcomes such as farmers participation in activities. As evaluations which only address issues

such as participation (level 3) would not provide any evidence of worth or merit of a program.

• ‘Level six’; changes in behaviour. Despite the fact that the higher level outcome of behavior

change is harder to attribute to the program activities alone, evaluation of these programs

should make an attempt to provide evidence concerning the extent to which these programs

contribute toward behavioural change (which is after all the ultimate aim of extension

programs).

The review of evaluation in extension (Chapter 4) revealed that of the 50 evaluation case studies

analysed, only a handful addressed the outcomes above Bennett’s level 4 (see Section 4.7). By

contrast, it appears that the MSC model can provide evidence that would sit in Bennett’s Hierarchy

levels 5 and 6. Changes in skills, and attitudes are very much the subject matter of the stories

produced. There is also some attempt to monitor changes in behaviour and outcomes in terms of

changes in on-farm practice and profitability. As the stories of change are embedded in the context

around the event in question, along with reasons why the change occurred, the issue of attribution is

brought to the surface rather than ignored. Because of this, it is also possible to provide more

credible evidence for change occurring at level 6 (behaviour change). It would appear that one of

the strengths of this approach is that it can monitor change at a higher level of the outcome

hierarchy than would a traditional approach based on simple performance indicators, and provide

some evidence for casual attribution.

Nevertheless, without some further meta-analysis of the stories, the MSC process provides only

limited information that would help stakeholders make a full determination of the merit and worth

of the project being evaluated.

262

9.11 Summary: Extent to which the MSC Model adds to

the ‘Basket of Choices’ for Extension in Australia

In terms of adding to the existing ‘basket of choices’ for extension evaluation, the MSC process is

novel in being able to cope with the new genre of extension projects with diverse, context-specific

outcomes. In Chapter 3 I suggested that agricultural extension projects pose their own particular

challenges to the practice of program evaluation. An analysis of the changing nature of extension

and its organisations led to the conclusion that there is increasing demand for monitoring

approaches capable of fostering dialogue and learning. A new genre of process-oriented extension

projects also present challenges for evaluation that objectives-based evaluation approaches are

unable to meet. The review of evaluation practice in Australia (Chapter 4) revealed a range of

approaches to evaluation employed in extension evaluation, but that most of the studies were

limited to a summative assessment of project impact. The review showed a predominance of

objectives-testing approaches. There also appeared to be very little project monitoring and few

approaches able to foster long-term dialogue between multiple stakeholders. For these reasons, it

would appear that the MSC model has much to offer the ‘basket of choices’ for extension

evaluation.

However, it should be cautioned that it is unlikely that the MSC model will suffice as stand-alone

evaluation approach. It is best used in careful combination with other evaluation approaches. I also

suggested that some project contexts are more conducive to gaining positive benefit from this

evaluation model than others. In particular, it would seem that large, process-orientated projects,

with high levels of farmer involvement, could gain most from this process.

9.12 Improving the Contribution of the MSC Model

It has been shown how the MSC model meets some of the premises for good evaluation, but does

not comply with them all. Like other evaluation approaches, the MSC model has inherent bias and

weaknesses. I would suggest two strategies to those considering adopting the MSC model. Firstly,

improvements that could be made to the process itself, to reduce some of the bias and

methodological weakness. Secondly the approach could be combined with other complementary

approaches to evaluation.

263

The MSC model could be improved by:

• Ensuring that there is a process to formally include the lessons learned from the stories into the

program planning, both long-term and short-term.

• Revising the sampling system of collecting stories to ensure that they are collected from

antagonists (including those who chose not to participate in the project) as well as advocates.

• Periodic review of the stories en masse. Unanticipated, valued outcomes should be identified

from the process, and a further step should be added to quantifying the spread of some these

outcomes. This could help to develop a system of quantifying dynamic indicators.

• Ensuring that more regular feedback is provided to the project from the purchasers.

The MSC model has quite a different set of biases to conventional methods of monitoring and

evaluation. For this reason, it is a particularly good tool to use in combination with other methods

of evaluation. The sorts of evaluation demands being made on extension projects under the current

reforms in extension organisations mean that it is highly unlikely that any one method of evaluation

would be sufficient on its own. Thus, the MSC model should be seen as one of a number of

methods, carefully chosen to offset different biases and meet the numerous evaluation demands. I

suggest that evaluation approaches that would work well with the MSC model, would be those that

can provide:

• evidence of spread of the emergent outcomes

• evidence of the achievement of predetermined outcomes, (if these have been articulated)

• evidence of the ‘average’ experience of participants (or of subgroups of participants) as well as

exceptional outcomes

• information on the views of non-participating dairy farmers, and other ‘victims’ of the project

• information to help improve internal organisational and management processes

• improved knowledge with regard to the logic of the project intervention

• evidence of whether conjectured outcomes have been achieved, and in what situations, and

why.

One model that would appear to combine well with the MSC model is Pawson and Tilley’s (1997)

‘Realistic Evaluation’ model. This would depend, of course, on the evaluation needs of the specific

project to be evaluated. In the case of the Target 10 Project, for example, the current evaluation of

264

achievement of valued outcomes could be replaced by a realistic approach to evaluation, helping to

meet the need for more evaluative information for accountability purposes. The following

discussion presents one way that this marriage could occur.

The MSC process could provide a ‘search mechanism’ for identifying conjectured outcomes and to

tap into the ‘folk wisdom’ of the project stakeholders. This information could then built into an

initial program theory model, in conjunction with the stakeholders. The construction of a program

theory model based on critical incidents could further help those constructing the model to

understand the underlying logic of the program. If, during the MSC process, some stories were

collected from antagonists of project practices, then it might also be possible to construct some

conjectured CMO configurations of unexpected negative outcomes.

The conjectured CMO configurations (developed from the MSC process) could then be used as the

starting point for a ‘Realistic Evaluation’. In this process, the conjectured CMOs would be refined,

and the achievement of these CMOs tested against empirical, and possibly quantitative data. It

might be beneficial at this point to employ external evaluators to conduct this part of the evaluation.

This would help provide evaluative information with high validity for project purchasers.

The MSC process could counterbalance some of the criticisms discussed in Section 5.5; i.e., that

‘Realistic Evaluation’ does not provide processes to:

• capture monitoring information

• address values (Julnes and Mark, 1998)

• involve stakeholders adequately in the evaluation (Leviton, 1994)

• incorporate more utilisation (Patton 1997) – the MSC model does this through the inclusion of

purchasers and other stakeholders actively in the evaluation process

• address unanticipated outcomes.

A realistic approach on the other hand, can counter criticism that the MSC process does not provide

generalisable findings, is not a theory-guided approach, and does not produce ‘objective’ evidence

of achievement of predetermined outcomes. In addition, potential outcomes of the MSC process,

such as enhanced communication and dialogue, would complement the sort of evaluative

information produced theory-driven evaluation processes and perhaps make it more palatable to

participatory or community based projects.

265

CHAPTER 10: CONCLUSIONS

The aim of this thesis was to develop a model of evaluation that had the potential to meet the needs

of agricultural extension projects in Australia. To develop this novel approach to evaluation, five

contemporary models of program evaluation were examined against a set of criteria. These criteria

were associated with the extent to which the model was (a) a novel approach, (b) could help

address the needs of evaluation at a national level, and (c) could meet the needs of one particular

case study project. From this analysis, the Most Significant Change model was selected (Davies

1996). It was a highly novel approach in that it had never been implemented in Australia before,

and was based on the unusual methodology of the collection and participatory selection of stories of

significant change.

Through the implementation of a modified version of the MSC model, I found that the process was

able meet the articulated needs of the stakeholders of one large extension project. This was

ultimately verified by their commitment to continue with the MSC process after a 12-month trial

was complete. The MSC model combined well with this extension project’s existing evaluation

practices that were objectives-based. The MSC model offered a process to encourage reflective

practice of the staff and collaborators, and helped them to make sense of unanticipated impact and

of each others’ values in relation to these impacts. The MSC model also helped draw staff, farmers

and other collaborators more centrally into the evaluation process.

The lessons learned from the implementation of the MSC model across the case evaluation were

considered in relation to the evaluation of agricultural extension projects more broadly. While the

case project cannot be considered representative of all large extension projects, the methods applied

in the meta-evaluation provided information concerning the sort of project contexts in which the

MSC model is likely to bring about desirable outcomes. In general, the MSC model showed

potential to address some of the evaluation needs of extension projects at a national level. It is

ideally suited to provide performance information that can foster organisational learning and

enhance communication in a context of organisational complexity and value pluralism. The stories

contain a wealth of information, both factual and tacit and the interpretations of the stories reveal

what stakeholders value and do not value in terms of impacts of project activities.

266

However, the MSC model does not offer processes to meet all the evaluation needs that were

identified at a national level. It did not provide the sort of data that is being demanded under the call

for greater accountability to clients and purchasers: it was not designed to provide the sort of data

generally accepted as ‘highly objective’. The MSC model as applied privileges the voice of people

internal (or close) to the project – those most likely to be advocates of the project. The MSC process

did not provide indication of the spread of adoption of technologies across the farming population:

the model is designed to capture ‘remarkable events’ rather than the average experience of farmers.

The MSC model aims to encourage learning between project stakeholders and to ‘improve’ the

project, rather than ‘prove’ what the project has achieved. Consequently, if used in isolation, the

MSC is unlikely to satisfy all stakeholders’ demands for evaluation.

In terms of adding to the existing ‘basket of choices’ for extension evaluation, the MSC process is

novel in that it is able to cope with the new genre of participatory extension projects. Where

projects have few or no pre-determined outcomes and produce diverse and context-specific

outcomes, the MSC model may be able to play an important stepping-stone to summative

evaluation. The MSC process can be viewed as a ‘search mechanism’ for unexpected or

unanticipated outcomes. Where outcomes are not predetermined the function of ‘searching’ for

outcomes could be very important. The MSC model could also be particularly relevant for

evaluating highly participatory projects. It can accommodate a wide range of participants in the

evaluation process itself. Highly participatory projects require participatory evaluation; approaches

that allow the stakeholders and beneficiaries to have a say in which changes are important, and

which changes should be measured. The MSC process has potential to meet these requirements.

However, I found that the MSC model does not meet with all the premises for good evaluation that

were developed in Chapter 2. Firstly, it is not guided by questions developed from a carefully

constructed model of the theory that underpins the project being evaluated. The MSC model

involves collecting impact information in an almost ‘goal-free manner’ and then coming an

understanding about key areas of findings through a process of induction. As a result of the lack of

emphasis on drawing the key questions from an explicit program theory, there is little analysis of

the underlying mechanisms that are triggered by the project.

Secondly, while the process may capture a range of unexpected outcomes perceived to be important

by the project stakeholders, it does not capture unexpected or unintended outcomes unless they are

deemed ‘significant’ by those involved in the process. This appears to be a serious omission to the

267

evaluation process. Thirdly, it can be argued that the MSC process was not socially /politically just,

in that it favoured the inclusion of some stakeholders over others. The MSC process is clearly

embedded within the discourse of the project staff and committee members. However, unlike other

research approaches, the bias itself if offered as another source of information about the

organisational capacity of the project.

Fourthly, the sampling technique of the MSC model is selective rather than inclusive. It will not

provide information on the ‘average condition’ of a farmer who attends the programs, but rather

will provide information about the exceptional circumstances, particularly successful

circumstances. Some would argue that such purposive sampling is not an appropriate technique to

produce reliable information to enable judgements to be made about the performance of the

program. Nevertheless, in qualitative research, purposive sampling is seen as a legitimate form of

data inquiry.

Finally, while stories of individual farmer outcomes were judged for their merit and worth by

stakeholders, there was no point at which the evaluators (stakeholders in this case) made an overall

determination of the merit and worth of the Target 10 Project. One of the strengths of this approach

is that it can monitor change at a higher level of the outcome hierarchy than would a traditional

approach based on simple performance indicators, and provide some evidence for casual attribution.

Nevertheless, without some further meta-analysis of the stories, the MSC process provides only

limited information that would help stakeholders make a full determination of the merit and worth

of the project being evaluated.

It has been shown how the MSC model meets some of the premises for good evaluation, but does

not comply with all. Like other evaluation approaches, the MSC model has inherent bias and

weaknesses. Two strategies were proposed to those considering adopting the MSC model. Firstly,

improvements could be made to the process itself, to reduce some of the bias and methodological

weakness. Secondly, the approach could be combined with other complementary approaches to

evaluation. The MSC model could be improved by including processes to formally include the

lessons learned into the program planning, ensure that the views of antagonists are captured,

periodically review the stories en masse, and ensure that regular feedback is provided to the project

from the funders.

268

As the MSC model presents quite a different set of biases to conventional methods of monitoring

and evaluation, it represents a particularly good tool to use in combination with other methods of

evaluation. The sorts of evaluation demands being made on extension projects under ‘new public

management’ mean that it is highly unlikely that any one approach to evaluation would be sufficient

on its own. I suggest that the MSC model should be strategically combined with other evaluation

approaches selected to offset the bias and to meet the other demands of evaluation. A model such as

Pawson and Tilley’s realistic evaluation might be particularly well suited for use alongside the MSC

process. Nevertheless, the MSC model would be more useful to extension if it was modified to

overcome at least some of its inherent biases.

The MSC model is an important contribution to the ‘basket of choices’ for extension evaluation, but

it does not provide an overarching solution to the myriad of evaluation demands being felt by

extension agents in Australia. I suggest that those evaluating extension projects should firstly come

to understand the evaluation needs of the project stakeholders, and then develop a carefully crafted

collection of evaluation approaches and methods to meet these requirements. For some projects, the

MSC model may provide an important new component of their evaluation bricolage. In particular,

it would seem that large, process-orientated projects, with high levels of farmer involvement, could

gain much from this process.

In this thesis, a novel evaluation approach has been empirically tested against standards for good

evaluation, and against evaluation needs. This type of research is extremely rare in the literature of

program evaluation, which is dominated by texts prescribing one evaluation model over others. I

suggest that meta-evaluation research conducted on other evaluation models could greatly assist

evaluators in the strategic selection and combination of evaluation approaches to meet the needs of

particular extension projects. In conducting meta-evaluation it seems valuable to adopt an empirical

and theory guided approach. Experience in this study suggests that Pawson and Tilley’s (1997)

Realistic Evaluation could provide a good model to use for this purpose, although more research is

needed to fine-tune such meta-evaluation techniques.

269

REFERENCES

Alasuutari, P. (1995), Researching Culture: Qualitative Method and Cultural Studies, Sage, London.

Alford, J. and O’Neill, D. (1994), The Contract State: Public Management and the Kennett Government,

Centre of Applied Social Research, Deakin University, Melbourne.

Alkin, M. C. (1972), ‘Evaluation comment: Wider context goals and goal-based evaluators’, Journal of

Educational Evaluation, 3, no.4, 10-11.

Alkin, M. C., Daillak, R. and White, P. (1979), Using Evaluations: Does Evaluation Make a Difference? Sage

Publications, Beverly Hills, CA.

Allen, W. J. (1998), ‘Towards improving the role of evaluation within natural resource management R&D

programs: The case for learning by doing’, Canadian Journal of Development Studies, 18 Special

Issue on Results-Based Evaluation, 629-643.

Anderson, F. M. and Dillon, J. L. (1985), ‘Farming systems research in the IARCs and other international

groups’, in Agricultural Systems Research for Developing Countries,vol. 11 (ed Remenyi, J. V.)

ACIAR Proceedings.

Appleyard, L. (1996a), ‘Target 10 Project evaluation’, in Agriculture and Food Initiative: Dairying.

Evaluation Report: Completed Projects, (ed Lew, M.) Department of Agriculture, Energy and

Minerals, Melbourne, 31-40.

Appleyard, L. (1996b), ‘APPRAISAL, A spreadsheet for evaluating returns to agricultural research’, Working

Paper, Performance Evaluation Division, Economics Branch, Department of Natural Resources and

Environment, Melbourne.

Argyris, C. and Schon, D. A. (1996), Organizational Learning II, Addison-Wesley Publishing Company,

Reading, Massachusetts.

Australasian Evaluation Society (1997), Membership Directory, Australasian Evaluation Society, Curtin,

Western Australia.

Bardsley, B. (1997), 3 Par Court, Bacchus Marsh, Victoria 3340.

270

Bardsley, J. B. (1981), Farmers’ assessment of information and ITS sources; An investigation using

interactive computer techniques, PhD thesis, Department of Agriculture, Horticulture and Forestry,

University of Melbourne.

Baume, P. S. (1977), ‘The politician’s role in the evaluation process’, Proceedings of the Conference of the

Australian and New Zealand Society for Epidemiology and Research in Community Health,

University of Adelaide.

Bawden, R. J. (1991), ‘Systems thinking and practice in agriculture’, Journal of Dairy Science, 74, 2362-

2373.

Bebbington, A. J., Merrill-Sands, D. and Farrington, J. (1994), ‘Farmer and community organisations in

agricultural research and extension: functions, impacts and questions’, ODI Agricultural

Administration (Research and Extension) Network paper, no. 47.

Beckingsale, D. (1997), Natural Resources and Environment, 8 Nicholson Street, East Melbourne, Victoria,

3002.

Beilin, R. (1998), Looking for Landcare: The landscape and the family farm, PhD thesis, RMIT University,

Melbourne.

Bell, J. (1983), Contemporary Social Welfare, Macmillan, New York.

Bennett, C. F. (1975), ‘Up the hierarchy’, Journal of Extension, March / April, 6-12.

Bennett, C. F. (1977), ‘Analysing impacts of extension programs’, Extension service, US Department of

Agriculture, Washington D.C.

Bickman, L. (1987), Advances in Program Theory, Jossey-Bass, San Francisco, CA.

Blank, S. C. (1999), The End of Agriculture in the American Portfolio, Quorum Books, Westport, USA.

Boje, D.-M. (1991), ‘The storytelling organization: A study of story performance in an office-supply firm’,

Administrative Science Quarterly, 36, no.1, 106-126.

Boomsma, J. (2000), Target 10 Dairy Extension Project, Natural Resources and Environment, RMB 2460,

Ellinbank, Victoria, 3821.

Boomsma, J., McDonald, B. and Walton, M. (1996), ‘Target 10: A report on the first three years’, Internal

report, Agriculture Victoria, NRE, Warragul.

271

Campbell, A. and Junor, R. (1992), ‘Land management and extension in the ‘90s: Evolution or

emasculation?’ Australian Journal of Soil and Water Conservation, 5, no. 2, 16-23.

Campbell, D.T. (1969), ‘Variation and selective retention in socio-cultural evolution’, General Systems, 16,

69-85.

Campbell, D. T. (1969b), ‘Reforms as experiments’, American Psychologist, 24, 409-429.

Campbell, D. T. (1991), ‘Methods for the experimenting society’, Evaluation Practice, 12, 223-260.

Campbell, K. O. (1980), Australian Agriculture: Reconciling Change and Tradition, Longman Cheshire,

Melbourne.

Caracelli, V. J. and Greene, J. C. (1997), ‘Crafting mixed-method evaluation designs’, in New Directions for

Program Evaluation, vol. 74: The Challenges and Benefits of Integrating Diverse Paradigms,

Advances in Mixed-Method Evaluation, (eds, Caracelli, V. J. and Greene, J. C.) Jossey-Bass, San

Francisco, CA.

Caron, D. J. (1993), ‘Knowledge required to perform the duties of an evaluator’, Canadian Journal of

Program Evaluation, 8, 59-78.

Cary, J. W. and Salmon, P. W. (1976), ‘Delphi and participatory planning: Focussing the planning process in

an agricultural service organisation’, Agricultural Extension Research Unit, School of Agriculture

and Forestry, University of Melbourne,

Chambers, R. (1983), Rural Development: Putting the last first, Longmann, Harlow, UK.

Chambers, R., Pacey, A. and Thrupp, L. A. (1991), Farmer First, Intermediate Technology Publications,

London.

Chen, H. T. (1990), Theory-Driven Evaluations, Sage Publications, Newbury Park, US.

Clark, R. A., Bourne, G. F., Cheffins, R. C., Esdale, C. E., Filet, P. G., Gillespie, R. C. and Graham, T. W. G.

(1996), The sustainable beef production systems project: Beyond awareness to continuous

improvement, Project Report Series, Queensland Department of Primary Industries, Brisbane.

Cook, D. L. (1966), Program Evaluation and Review Technique Applications in Education, Government

Printing Office, Washington D.C.

Cook, T. D. and Campbell, D. (1979), Quasi-Experimentation: Design Analysis Issues for Field Settings,

Rand McNally, Chicago.

272

Coutts, J. A. (1997), ‘Changes in extension: An Australian perspective. The coming of age of extension’,

Proceedings of 2nd Australasia Pacific Extension Network (APEN) Conference, Albury, New South

Wales, APEN, 22-32.

Cronbach, L. J. (1963), Evaluation for Course Improvement, Harper & Row, New York.

Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D. F.

and Weiner, S. S. (1980), Toward Reform of Program Evaluation: Aims, Methods, and Institutional

Arrangements, Josey-Bass, San Francisco.

Cunningham, J. B. (1993), Action Research and Organisational Development, Praeger, Westport,

Connecticut.

Curtis, A. and De Lacy, T. (1994), ‘Landcare: Does it make a difference?’ Charles Sturt University, Albury,

New South Wales.

Dart, J. J. (1997), ‘Facilitating tribal farmers to identify change to their farming systems using participatory

farming systems models’, Evaluation Journal of Australasia, 9, no.1&2, 31-37.

Dart, J. J. (1999a), ‘A story approach for monitoring change in an agricultural extension project’, Proceedings

of Conference of the Association for Qualitative Research, Melbourne, published on the world

wideweb: http://www.latrobe.edu.au/www/aqr/offer/papers/JDart.htm.

Dart, J. J. (1999b), Evaluating the ‘kinder surprise model’: The evaluation of the organisational service

delivery model of the Target 10 Dairy Extension Project, Evaluation report, Target 10 Project,

Natural Resources and Environment, Warragul, Victoria.

Dart, J. J., (ed) (1999c), ‘Target 10 evaluation stories, May 1998-May 1999’, Evaluation report, Target 10

Project, Department of Natural Resources and Environment, Warragul, Victoria

Dart, J. J., Petheram, R. J. and Straw, W. (1998), Review of Program Evaluation in Agricultural Extension,

Rural Industries and Research Development Corporation, no. 98/136, Canberra.

Datta, L. (1994), ‘Paradigm wars: A basis for peaceful coexistence and beyond’, The qualitative-quantitative

debate: New perspectives, (eds Reichardt, C. S. and Rallis, S. F.) 53-70.

Datta, L., E. (1997), ‘A pragmatic basis for mixed-method designs’, New Directions for Program Evaluation,

vol. 74: Advances in Mixed Method Evaluation: The Challenges and Benefits of Integrating Diverse

Paradigms (eds Green, J. C. and Caracelli, V. J.) Jossey-Bass, San Francisco, 33-46.

273

Davies, R., J. (1999), Centre for Development Studies, University College of Swansea, Singleton Park,

Swansea, SA2 8PP, Wales, UK.

Davies, R. J. (1996), An evolutionary approach to facilitating organisational learning: An experiment by the

Christian Commission for Development in Bangladesh, Centre for Development Studies, world wide

web publication: http://www.swan.ac.uk/cds/rd/ccdb.htm, Swansea, UK.

Davies, R. J. (1998), Order and Diversity: Representing and Assisting Organisational Learning in Non-

Government Aid Organisations, PhD thesis, The Centre for Development Studies (CDS), University

of Wales, Swansea, UK.

De Vaus, D. A. (1995), Surveys in Social Research, Allen and Unwin, St Leonards, New South Wales.

Denzin, N. K. and Lincoln, Y. S. (eds) (1998), Collecting and Interpreting Qualitative Materials, Sage,

Thousand Oaks, CA.

Department of Finance (1991), Handbook of Cost-benefit Analysis, AGPS, Canberra.

Department of Health (1993), Code of practice for the commissioning and management of research and

development, HMSO, London.

Dillon, J. L. (1965), ‘Farm management in Australia as an academic discipline’, Review of Marketing and

Agricultural Economics, 33,175-189.

DRDC (1994), DRDC- Funded Extension: Programs in Victoria, New South Wales and Tasmania, by Falvey

Consulting, Dairy Research and Development Corporation (DRDC), Melbourne.

Dunn, T. (1996), ‘Changing paradigms for farmers-researcher-extensionist relationships: Exploring methods

and theories of farmer participation in research’, European Journal of Agricultural Education and

Extension, 3, no.3, 167-181.

Eisner, E. W. (1985), The Art of Educational Evaluation: A Personal View, Falmer Press, London.

Engel, P. G. H. and Van den Bor, W. (1995), ‘Agricultural education from a knowledge systems perspective:

from teaching to facilitating joint inquiry and learning’, European Journal of Agricultural Education

and Extension, 1, no.4, 1-23.

Estrella, M. and Gaventa, J. (1998), Who Counts Reality? Participatory Monitoring and Evaluation: A

Literature Review, Institute of Development Studies, London.

274

Farquhar, R. N. (1961), Comparative extension education in anglo-saxon countries with particular references

to Australia, PhD thesis, Cornell University, US.

Farrington, J. and Nelson, J. (1997), ‘Using logframes to monitor and review farmer participatory research’,

Agricultural Research and Extension Network, no.73.

Feinstein, O. N. (1998), ‘Book review: Realistic Evaluation’, Evaluation, 4, no.2, 243-246.

Fisher, R. A. (1956), Statistical Methods and Scientific Inference, Oliver and Boyde, Edinburgh.

Fishman, D. B. (1992), ‘Postmodernism comes to program evaluation: A critical review of Guba and

Lincoln’s Fourth Generation Evaluation’, Evaluation & Program Planning, 15, no.3, 263-270.

Foale, M. A. (1997), ‘Management of dryland farming systems in North Eastern Australia. How can scientists

impact on farmer’s decision making?’, Agricultural Science, 10, no., 34-37.

Freebairn, J. W. (1989), ‘How to cut public spending’, Policy, Autumn.

Friere, P. (1972), Pedagogy of the Oppressed, Penguin, Harmondsworth, UK.

Geertz, C. (1973), The Interpretation of Cultures, Basic Books, USA.

Giddens, A. (1998), The Third Way: The Renewal of Social Democracy, Polity Press, Cambridge, UK.

Guba, E. G. and Lincoln, Y. S. (1981), Effective Evaluation, Jossey-Bass, San Francisco.

Guba, E. G. and Lincoln, Y. S. (1989), Fourth Generation Evaluation, Sage, Newbury Park, US.

Guerin, L. J. and Guerin, T. F. (1994), ‘Constraints to the adoption of innovation in agricultural research and

environmental management: a review’, Australian Journal of Experimental Agriculture, 34, 549-

571.

Guijt, I. and Gaventa, J. (1998), Participatory monitoring & evaluation: Learning from change, Policy

Briefing, Institute of Development Studies, Brighton,UK.

Hamilton-Smith, N. and Hopkins, M. (1998), ‘Theory-driven interviewing: from theory into practice’,

Bulletin de Methodologie Sociologique, 60, 80-102.

Harre, R. (1986), Varieties of realism: a rationale for the natural sciences, Blackwell, Oxford, UK.

275

Hart, E. and Bond, M. (1995), Action Research for Health and Social Care: A Guide to Practice, Open

University Press, Buckingham, UK.

Hood, C. (1991), ‘A public management for all seasons?’ Public Administration, 69, no.1, 3-19.

House, E. R. (1978), ‘Assumptions underlying evaluation models’, Educational Researcher, 7, no.3, 4-12.

House, E. R. (1991), ‘Realism in research’, Educational Researcher, 20, no., 2-9.

Hynes, H. J. (1963), ‘Opening address’, Proceedings of the Australian Agricultural Extension Conference,

Melbourne, 4-6.

Jiggins, J. (1995), ‘Development impact assessment: Impact assessment of aid projects in non-western

countries’, in Environmental and Social Impact Assessment, (Eds, Vanclay, F. and Bronstein, D. A.)

Wiley and Sons, Chichester, New York.

Julnes, G. and Mark, M. M. (1998), ‘Evaluation as sensemaking: knowledge construction in a realist world’,

in New Directions for Evaluation, vol. 78: Realist Evaluation: An Emerging Theory in Support of

Practice (Eds, Henry, G. T., Julnes, G. and Mark, M. M.) Jossey-Bass Publishers, San Francisco, 33-

53.

Julnes, G., Mark, M. M. and Henry, G. T. (1998), ‘Promoting realism in evaluation: Realistic evaluation and

the broader context’, Evaluation, 4, no.4, 483-504.

Kaplan, A. (1964), The Conduct of Inquiry, Chandler, San Francisco.

Kazi, M. A. F. (forthcoming), ‘Realist evaluation’, in The Blackwell Encyclopaedia of Social Work, (ed

Davies, M.) Blackwell, Oxford.

Kefford, B. (1999), Department of Natural Resources and Environment, 8 Nicholson Street, East Melbourne,

Victoria 3002.

Kolb, D. A. (1984), Experiential Learning: Experience as a Source of Learning and Development,

Englewood Cliffs, New Jersey.

Lane, J-E. (1997), ‘Public sector reform in the nordic countries’, in Public Sector Reform: Rationale, Trends

and Problems, (ed Lane, J-E.) Sage, London, 188-208.

Lasswell, W. (1936), Politics: Who Gets What, When, How, McGraw Hill, New York.

Lazerfield, P. F. and Rosenburg, M. (eds) (1955), The Language of Social Research, Free Press, Glencoe, IL.

276

Lees, J. W. (1990), ‘More than accountability’, Rural Development Centre, University of New England,

Armidale, New South Wales.

Leviton, L. C. (1994), ‘Program theory and evaluation theory in community-based programs’, Evaluation

Practice, 15, no.1, 89-92.

Lewin, K. (ed.) (1948), Resolving Social conflicts: Selected papers on group dynamics, Harper and Brothers,

New York.

Louis, M. R. (1980), ‘Surprise and sensemaking: What newcomers experience in entering unfamiliar

organizational settings’, Administrative Science Quarterly, 25, 226-251.

Macadam, R. D. (1996), ‘Developments in agricultural industries research and development in Australia with

particular reference on extension and the relevance of systematic developments as a response to the

current situation’, Proceedings of the Asia Pacific Seminar on Education for Rural Development,

Obihiro, Japan.

Madaus, G. F., Scriven, M. S. and Stufflebeam, D. L. (1983), Evaluation Models, Kluwer-Nijhoff Publishing,

Boston.

Manicas, P. T. and Secord, P. F. (1983), ‘Implications for psychology of the new philosophy of science’,

American Psychologist, 3, no., 399-413.

Marsh, S. P. and Pannell, D. J. (1997), ‘The changing relationship between private and public sector

agricultural extension in Australia’, Proceedings of the 2nd Australasia Pacific Extension Network

(APEN) Conference, Albury, New South Wales, APEN, 350-357.

Martin, S. and Sanderson, I. (1999), ‘Evaluating public policy experiments: Measuring outcomes, monitoring

processes or managing pilots?’ Evaluation, 5, no.3, 245-258.

McCown, R. L. (1991), ‘Research in a farming systems framework’, in Dryland Farming: A Systems

Approach, an Analysis of Dryland Agriculture in Australia (eds Squires, V. R. and Tow, P.G.),

Oxford University Press, Melbourne, 242-249.

McDonald, B. (1999), Natural Resources and Environment, RMB 2460, Ellinbank, Victoria, 3821.

McDonald, B. and Kefford, B. (1998), ‘Exploring a new approach to evaluation’, Proceedings of

Performance Measures for the Public Sector, Sydney, Institute for International Research, Sydney.

277

Morrison, D. (1993), ‘Square pegs in round holes: perspectives of cost-benefit gained from evaluation of

Western Australia Department of Agriculture activities’, Gains for Grains, 4: Benefit-cost Methods

in Research Evaluation, Canberra.

Nelson, C., Treichler, P. A. and Grossberg, L. (1992), ‘Cultural studies’, in Cultural Studies, (eds Grossberg,

L., Nelson, C. and Treichler, P. A.) Routledge, New York.

Nelson, R. and Winter, S. G. (1982), An Evolutionary Theory of Economic Change, The Belknap Press of

Harvard University Press, Cambridge, Mass.

NRE (1997), ‘Standard for project development and delivery’, by the Agricultural Industries Team, Natural

Resources and Environment (NRE), Victoria, Melbourne.

Oakley, P., Pratt, B. and Clayton, A. (1998), Outcomes and Impact: Evaluating Change in Social

Development, INTRAC, Oxford.

O’Brien, G. and Hepworth, G. (1993), ‘Evaluation of a pasture management program for dairy farmers in

West and South Gippsland’, Department of Agriculture, Melbourne.

O’Faircheallaigh, C., Wanna, J. and Weller, P. (1999), Public Sector Management in Australia: new

challenges new directions, MacMillan Education Australia, Melbourne.

Owen, J. M. (1993), Program Evaluation, Forms and Approaches, Allen and Unwin, New South Wales,

Australia.

Patton, M. Q. (1990), Qualitative Evaluation and Research Methods, Sage, Newbury Park,US.

Patton, M. Q. (1997), Utilization Focused Evaluation, Sage, Thousand Oaks, US.

Patton, M. Q. (1999), ‘Book review of realistic evaluation’, American Journal of Evaluation, 20, no.2, 385-

388.

Patton, M. Q. (2000), Re: Ohio’s Evaluation Conference, description of workshop run by Micheal Patton,

GOVETAL (an international E-mail discussion list on public sector program evaluation, Thursday

April 2000).)

Pawson, R. and Tilley, N. (1997), Realistic Evaluation, Sage, London.

Petheram, R. J. (2000), School of Forestry, University of Melbourne, Water Street, Creswick, Victoria, 3363.

Petheram, R. J. and Clark, R. A. (1998), ‘Farming systems research: relevance to Australia’, Australian

Journal of Experimental Agriculture, 38, 101-15.

278

Polanyi, L. (1985), Telling the American story: A structural and cultural analysis of conversational

storytelling, Ablex, Norwood, NJ.

Pollitt, C. (1997), ‘Evaluation and the New Public Management: An international perspective’, Proceedings

of the Australasian Evaluation Society Conference (AES), Adelaide, AES, 17-23.

Popper, K. (1959), The Logic of Scientific Discovery, Hutchinson, London.

Pretty, J. N. (1995), ‘Participatory learning for sustainable agriculture’, World Development, 23, no.8, 1247-

1263.

Pretty, N. P., Guijit , I. and Scoones, I. (1995), Participatory Learning and Action: a Trainer’s Guide,

International Institute for Environment and Development, London.

Prince, G. (1973), A Grammar of Stories, Mouton, The Hague.

Pusey, M. (1996), Economic Rationalism in Canberra, Cambridge University Press, Cambridge, UK.

Reason, P. (ed) (1988), Human Inquiry in Action: Developments in new paradigm research, Sage, London.

Rhoades, R. and Boothe, R. (1982), ‘Farmer-back to farmers: A model for generating agricultural

technology’, Agricultural Administration, 11,127-137.

Riches, J. R. H. (1973), Evaluation of a farm business management extension program, Working Papers,

Glenormiston Agricultural College, Noorat, Victoria.

Riessman, C. K. (1993), Narrative Analysis, Sage, Newbury Park.

Rist, R. C. (1997), ‘Evaluation and organisational learning: Some international perspectives’, Proceedings of

the Australasian Evaluation Society (AES), Adelaide, AES, 24-38.

Roberts, G. M. O. (1997), Action researching my practice as a facilitator of experiential learning with

pastoralist farmers in central West Queensland, PhD thesis, University of Western Sydney,

Richmond, New South Wales.

Roberts, K. C. (1998), A coexistive model of evaluation using Landcare as a case study, PhD thesis, School of

Natural and Rural Systems Management, University of Queensland, Gatton.

Rogers, E. M. (1962), Diffusion of Innovations, Free Press, New York.

279

Rogers, P. R. (1996), Evaluating approaches to program evaluation: A framework and its application to a

meta-evaluation of Patton’s Utilisation-Focused Approach, PhD thesis, Faculty of Education,

University of Melbourne.

Röling, N. (1988), Extension Science, Cambridge University Press, Cambridge.

Röling, N. and Jiggins, J. (1987), ‘Developments in European agricultural extension practices and the

implications for Australia - present and future’, Proceedings of the QDPI Extension Symposium,

Brisbane, Queensland Department of Primary Industries.

Rossi, P. H., Freeman, H. E. and Write, S. R. (1979), Evaluation: A systematic Approach, Sage publications,

Beverly Hills, CA.

Russell, D. B., Ison, R. L., Gamble, D. R. and Williams, R. K. (1989), A critical review of rural extension

theory and practice, Australian Wool Corporation, Sydney.

Scriven, M. (1967), ‘The methodology of evaluation’, Curriculum Evaluation, American Educational

Research Association Monograph Series on Evaluation, vol. 1 (ed Stake, R. E.), Rand McNally,

Chicago, 39-83.

Scriven, M. (1976), ‘Evaluation bias and its control’, Evaluation Studies Review Annual, vol. 1 (ed Glass

G.V.) Sage, Beverly Hills, 101-118.

Scriven, M. (1980), The Logic of Evaluation, Edgepress, Inverness, CA.

Scriven, M. (1986), ‘New frontiers of evaluation’, Evaluation Practice, 7, 7-44.

Scriven, M. (1991), Evaluation Thesaurus, Sage, Newbury Park,US.

Scriven, M. (1994), ‘The fine line between evaluation and explanation’, Evaluation Practice, 15, no. 1, 75-77.

Sechrest, L., West, S. G., Phillips, M., Redner, R. and Yeaton, W. (1979), ‘Some neglected problems in

evaluation research: Strength and integrity of treatments’, in Evaluation Studies Review Annual, vol.

4 (ed Sechrest & Associates), Sage, Beverly Hills, 15-35.

Shadish, W. R. (1992), ‘Theory-driven meta-evaluation’, in Using Theory to Improve Program and Policy

Evaluations, (eds Chen, H-T. and Rossi, P. H.), Greenwood Press, New York.

Shadish, W. R., Cook, T. D. and Leviton, L. C. (1995), Foundations of Program Evaluation, Sage, CA.

Shadish, W. R. and Epstein, R. E. (1987), ‘Patterns of program evaluation practice among members of

Evaluation Research Society and Evaluation Network’, Evaluation Review, 11, 555-590.

280

Sharp, C. and Lindsay, A. (1992), ‘An interim history of program evaluation in Australia and New Zealand

and the Australasian Evaluation Society’, Special Keynote Paper, Proceedings of Australasian

Evaluation Society (AES) International Evaluation Conference, AES, Melbourne.

Shaw, G., Brown, R. and Bromiley, P. (1998), ‘Strategic stories’, Harvard Business Review, May-June, 41-

50.

Shaw, H. (2000), Natural Resources and Environment, Bendigo, Cnr Midland Highway and Taylor Street,

Epsom, Victoria 3351.

Smith, M. F. (1989), Evaluability Assessment: A Practical Approach, Kluwer Academic, Boston.

Smith, N. L. (1994), ‘Evaluation models and approaches’, in International Encyclopedia of Education, vol. 4

(eds Husen, T. and Postlethwaite) Elsevier Science, Oxford, 2101-2109.

Spencer, A. (n.d.), ‘Improving the usefulness of the pigpulse report, an evaluation’, Rural Extension Centre,

University of Queensland, Gatton.

Spradley, J. P. (1979), The Ethnographic Interview, Holt / Rinehart and Winston, New York.

Sri Pathmanathan, C. (1978), Agricultural research project abstracts, Publications in Agricultural Extension,

University of Melbourne, School of Agriculture and Forestry, Melbourne.

Stake, R. E. (1967), ‘The countenance of educational evaluation’, Teachers College Record, 68, 523-40.

Stake, R. E. (1973), ‘Program evaluation: particularly responsive evaluation’, in Evaluation Models, (Eds,

Madaus, G. F., Scriven, M. S. and Stufflebeam, D. L.) Kluwer-Nijhoff, Boston.

Stanfield, R. B. (1997), The Art of Focused Conversation: 100 ways to Access group wisdom, A TOP

Method, Institute of Cultural Affairs, Melbourne.

Stewart, L. (1994), ‘A review of the monitoring and evaluation of property management planning in South

East Queensland’, Department of Primary Industries, Queensland.

Stufflebeam, D. L. and Webster, W. J. (1981), ‘An analysis of alternative approaches to evaluation’, in

Evaluation Studies Review Annual, vol. 6 (Eds, Freeman, H. E. and Solomon, M. A.) Sage, Beverly

Hills, 70-85.

Tashakkori, A. and Teddlie, C. (1998), Mixed Methodology: Combining Qualitative and Quantitative

Approaches, Sage, Thousand Oaks, Country.

281

The Virtual Consulting Group (1999), ‘Review of extension and human resource development for the

Australian dairy industry’, Evaluation report, Melbourne.

Toulemonde, J., Fontain, C., Laudren, E. and Vincke, P. (1998), ‘Evaluation in partnership: Suggestions for

improving their quality’, Evaluation, 4, no. 2, 171-189.

Tyler, R. W. (1967), ‘Changing the concepts of educational evaluation’, in Perspectives of Curriculum

Evaluation, vol. 1 (ed Stake, R. E.) Rand McNally, New York.

Van den Ban, A. W. and Hawkins, H. S. (1988), Agricultural Extension, Blackwell Science, Oxford.

Van Dissel, H. (1986), ‘Editorial comment’, AREX Newsletter, 3, no.1.

Vanclay, F. (1994), ‘A crisis in agricultural extension’, Rural Society, 4, no.1, 10-14.

Vanclay, F. and Lawrence, G. (1995), The Environmental Imperative: Eco-social Concerns for Australian

Agriculture, Central Queensland University Press, Rockhampton, Queensland.

Wadsworth, Y. (1991), Everyday Evaluation on the Run, Action Research Issues Association (incorporated),

Melbourne.

Watson, A., Hely, R., O’Keefe, M., Cary, J., Clark, N. and Allen, J. (1992), ‘Review of field-based services in

the Victorian Department of Food and Agriculture’, Department of Natural Resources and

Environment,Victoria.

Weick, K. E. (1995), Sensemaking in Organisations, Sage, Thousand Oaks, CA.

Weinstein, D. and Weinstein, M. A. (1991), ‘Georg Simmel: Sociological flaneur bricoleur’, Theory, Culture

and Society, 8, 151-168.

Wholey, J. S. (1983), Evaluation and Effective Public Management, Boston, Little Brown.

Whyte, W. F. (1991), Participatory Action Research, Sage, Newbury Park, USA.

Williams, D. B. (1968), Agricultural Extension, Melbourne University Press, Melbourne.

Winston, J. (1997), Program for Public Sector Evaluation (PPSE), Faculty of Applied Science, Royal

Melbourne Institute of Technology (RMIT), PO Box 71, Bundoora, Victoria 3083.

Winston, J. A. (1991), ‘Linking evaluation and performance management, review paper’, Proceedings of

National Evaluation Conference, Adelaide, Australasian Evaluation Society, 598-607.

282

Woodhill, J., Trevethan, P., Frankenburg, J., Beaulieu, C. and West Hume Landcare Group (1994), A case

study undertaken with the West Hume Landcare Group as part of an international research project

on participatory watershed development, International Institute for Environment and Development,

Land and Water Resources Research and Development Corporation; and Greening Australia.

Woods, E., Moll, G., Coutts, J., Clark, R. and Irvine, C. (1993), Information exchange: a review, Land and

Water Resources Research and Development Corporation, Canberra.

Worthen, B. R., Sanders, J. R. and Fitzpatrick, J. L. (1997), Program Evaluation: Alternative Approaches and

Practical Guidelines, Longman, New York.

Wortman, P. M. (1983), ‘Evaluation research: A methodological perspective’, Annual Review of Psychology,

34, 223-60.

Zohar, D. and Marshall, I. (1994), The Quantum Society: Mind, physics and a new social vision, Flamingo,

London.

Zuurbier, P. J. P. (ed) (1983), De Rekaties Tussen Onderzoek, Voorlichting en de Boer, Ministry of

Agriculture and Fisheries, The Hague.

284

APPENDICES

285

Appendix 1 Interview guides

The following lists of questions were used as guides for the interviewer in the semi-structured

interviews.

1.1 Guide questions for focus group with regional co-ordinators

1. Why do we need evaluation?

2. You know about the approach I am using, what kind of information do you think it will gather

and how are we going be able to use it?

3. Sentence completion: The thing that I find most useful for Target 10 about the approach is:

4. Sentence completion: The thing that I am most worried about with this process is….

5. Imagine that a year has gone past, how will we know that this approach has been useful?

1.2 Guide questions for focus-group of purchasers after the round table

meeting

1. Was there anything about this process – this methodology that makes you feel uncomfortable?

2. Was there anything about the process that you felt was particularly encouraging?

3. What messages should we as a group send back to Target 10?

4. If anther project comes to me i.e., TOPCROP, FARMSMART, WOOLPRO. And they say we are

interested in this story approach – should we do it? How should I respond to them?

1.3 Guide for semi-structured interviews with ten key informants after the trial

1. Do you remember when the story approach was first introduced to you? What were your initial

responses?

2. How do you feel about it now?

3. In what ways, if any, has the story approach affected the project? In other words, what impact

has the story approach had on the project?

Probe: how have the stories been used?

Probe: how could the story approach help improve the project?

4. How has it/ or will it affect your ability to do your work?

5. What changes do you recommend to the approach?

6. What kind of priority should this process have in the project?

286

Appendix 2 Internet meta-evaluation questionnaire for staff

287

288

289

290

291

292

293

294

Appendix 3 Internet ethics declaration

An ethics declaration was placed on the internet site for those responding to the questionnaire to

inspect. A hypertext link was provided from the questionnaire site (appendix 2) to the page shown

below.

295

296

297

Appendix 4 letter detailing rights of informants

The following letter was provided to all those who where to participate in either focus group

discussions or semi-structured interviews.

Dr R J Petheram Ms Jessica Dart Department of Forestry Institute of Land and Food Resources Longerenong College, RMB 3000, Horsham, Vic 3401

Department of Forestry Institute of Land and Food Resources Royal Parade, University of Melbourne, Parkville, Vic 3052

Tel 03 53 622253 Tel : (03) 9344 8356, Fax 03 53 622213 Fax: (03) 9344 4665 Email: [email protected] Email [email protected]

Dear Sir/Madam, This letter is to inform you of your rights of participation and confidentiality during your interview led by Jessica Dart. Jessica Dart is a PhD student from the Institute of Land and Food Resources, University of Melbourne. Her supervisor there is Dr R. Petheram. The research project that Jessica is undertaking, alongside the Target 10 Project staff, is concerned with developing a tool for evaluating agricultural extension projects. This tool involves the collection of stories of change and has been implemented with the Target 10 Dairy Extension Project since May 1997. In May 1997 this research project was endorsed by all four Regional Committees and by the Statewide Executive Committee of the Target 10 Project. These committees agreed to participate in this process for a period of one-year (June 1998 to June 1999) and to any interviews that may be associated with assessing the tool in June 1999. The interview that you have been asked to attend is concerned with your experience of being involved with the story approach. You are under no obligation to participate, and even after agreeing to participate, you may still choose to withdraw from the interview at any stage. During the interview you will be asked to about the extent to which you found the story approach to be useful, and about any issues of concern that you might have. For the purposes of her research Jessica is interested in both negative and positive attitudes towards the approach and we maintain that all views are valid. On your agreement, the interviews will be taped by Jessica and later transcribed. If you agree to this, your words may be quoted in publications, but your identity will be disguised. Any quotations used will be sourced in terms of the position of the informant and the individual identity will be protected at all times. (e.g., Source: NRE employee). Confidentiality of data will be preserved subject to any legal requirements. We have a commitment to the confidentiality of your responses (subject to legal requirements) so that you are not identifiable personally or any comments you make are not traceable to you personally. To ensure this, the following protocols will be followed: there will be no discussions between the researcher and research participants concerning comments made by others at separate interviews transcripts of interviews will be identified or coded in a way that is only recognisable to

298

the researcher (names and farm locations will be stored separately from the interview transcripts). In this way anonymity will be preserved. When results are reported, the identity of participants will be obscured through the use of pseudonyms only identifiable by the researcher. The findings of these interviews will be published in the PhD thesis of Jessica Dart, and in other academic publications. However, you are more than welcome to check any quotations prior to publication, and are free to withdraw your comments at any time.

If you have any inquiries please do not hesitate to ring Jessica Dart, or Dr RJ Petheram. Yours sincerely

Jessica Dart DR R J Petheram

299

Appendix 5 First round of Delphi questionnaire

In the case evaluation while developing the ‘domains of change’ for the MSC model, the Delphi

process was followed to ensure that the views of the numerous stakeholders were considered.

Delphi is a form of interactive (postal) surveying that utilises an iterative questionnaire and

feedback and provides participants with an opportunity to revise earlier views based on the response

of other participants, until some desired level of consensus is reached (Cary and Salmon, 1976).

This appendix shows that first letter and survey that was sent to participants, Appendix 6 shows the

results of this round, and Appendix 7 provides questionnaire for the second round of the process.

________________________________________________________________________________

Question Sheet – to Establish the Main ‘Types’ of Change to Monitor in the Target 10 Project In order to focus the monitoring process, we are seeking your input to help us choose the most important ‘types’ of change on which to collect stories. In this monitoring approach 3-4 broad ‘types’ of change are usually agreed upon before the story collection begins. Please indicate on this form which types of change you think are most important to monitor in order to understand the impact of the Target 10 activities. Please bear in mind that this approach aims to track change across all the programs and to gain information on the total impact of Target 10 activities. Please answer the following questions by placing tick in the box of the option that you most closely agree with: I believe that changes in perceived quality of life of project participants would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other_____________________________________________________________________ I believe that changes in project participant skills (interpersonal, practical, coping) would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other_____________________________________________________________________

300

I believe that changes in the goals and aspirations of project participants would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other____________________________________________________________________ I believe that changes in farm productivity would be: π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other____________________________________________________________________ I believe that changes in total on-farm management practice would be : π an essential type of change to monitor π a useful type of change to monitor π OK to monitor π of no value to monitor π wasteful to monitor π Other____________________________________________________________________

Please indicate any other types of change that you feel should be monitored in order to understand the overall impact of Target 10 activities: ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Please also tick the appropriate boxes below: Which region are you from? Which group(s) do you participate in? a) Northern Irrigation Region ρ a) Target 10 Statewide Executive ρb) North East ρ b) Target 10 regional committee ρc) South West ρ c) Target 10 working groups ρd) Gippsland ρ d) Extension staff ρe) Other / not relevant ρ e) Purchaser group ρ

301

f) Other ρ

Are you? a) A farmer ρb) A University/TAFE provider ρc) A dairy factory representative ρd) An NRE employee ρe) An agribusiness representative ρf) Other___________________________ ρ

Thank you for completing this form. In 2-3 weeks time you will receive the results of this survey and will have a chance to re-cast your vote with consideration of what other stakeholders in the project consider should be monitored. If I have not received your reply within 10 working days I will assume that you chose not to participate at this stage.

302

Appendix 6 Results of the first round

Results Sheet In May a survey was sent to 150 of the Target 10 stakeholders. This sheet explains the results of this survey to which 101 people responded. 1. Results to the tick-box questions In the first mail-out you were asked to indicate which types of change you thought were the most important to monitor in order to understand the impact of the Target 10 activities. You were asked to answer the five questions by placing a tick in the box of the option that you most closely agreed with. For example: I believe that changes in perceived quality of life of project participants would be: π an essential type of change to monitor π a useful type of change to monitor π ok to monitor π of no value to monitor π wasteful to monitor The results of the first mail-out are presented in the graph below. It can be seen that more people thought that changes in on-farm management practice were essential to monitor than any other ‘type’ of change. The second most highly scored ‘type’ of change was changes in farm productivity and thirdly changes in project participants’ skills.

0%

25%

50%

75%

100%

Responses

On-farmmanagement

practice

Farm productivity Project participantskills

Goals andaspirations

Perceived quality oflife

OK

Useful

Essential

303

In addition to answering the tick-box questions, respondents were also encouraged to offer other ‘types’ of change that they felt should be monitored. The graph on this page shows the number of times each alternative ‘type’ of change was mentioned. On this graph only the ‘types’ of change that were mentioned more than twice are listed. The graph below shows that 13 respondents listed changes in profitability to be an important type of change to monitor. Five respondents also thought that changes in the way farmers make decisions were important to monitor.

In the response sheet you will be asked to make a second vote for the different ‘types’ of change that the project could monitor to understand the overall impact of Target 10. This time please consider the responses of the other Target 10 stakeholders while deciding how to vote.

You may have noticed that I have not included changes in ‘who is participating’ in the response sheet. This is because Target 10 already keeps records of how many people attend meetings and so forth.

0

5

10

15

Responses

Env

ironm

enta

lco

nditi

ons

Atti

tude

sto

war

dsto

T10

Dea

ling

with

chan

ge

Farm

erco

nfid

ence

Indu

stry

wid

ech

ange

Who

ispa

rtici

patin

g

Farm

erde

cisi

onm

akin

gsk

ills

Farm

prof

itabi

lity

304

Appendix 7 Second round of Delphi questionnaire

Question Sheet – to Establish the Main ‘Types’ of Change to Monitor in the Target 10 Project In order to focus the monitoring process, we are seeking your input to help us choose the most important ‘types’ of change on which to collect stories. In this monitoring approach 3 broad ‘types’ of change are usually agreed upon before the story collection begins. Once the ‘types’ of change are decided on, stories will be collected each month concerning this type of change. There will be an opportunity to collect stories concerning other types of change as well. This process of defining ‘types’ of change is being carried out in order to provide some focus to the process and to ensure that certain types of change are tracked every month. Please indicate on this form which types of change you think are most important to monitor in order to understand the impact of the Target 10 activities. Please bear in mind that this approach aims to track change across all the programs and to gain information on the total impact of Target 10 activities. Please consider the results of the first survey (yellow sheet) in making your decision. When we start the monitoring process we will track 3 main ‘types of change’. ******************************************************************************** Please choose 3 ‘types’ of change that you think are most important to monitor every month – place a tick in 3 boxes only.

Changes in on-farm management practice ρChanges in farm productivity ρChanges in project participant skills (interpersonal, practical, coping) ρChanges in goals and aspirations of project participants ρChanges in the quality of life of project participants ρChanges in farm profitability ρChanges in participants’ decision-making skills ρChanges in participants’ confidence ρChanges in participants’ ability to deal with changing industry conditions ρChanges in the industry in general ρChanges in attitudes towards Target 10 ρChanges in the environmental conditions on the farm ρOther_________________________________________________ ρOther_________________________________________________ ρ

Any comments that you would like to add about this process? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

305

Your name______________________________________________________________

Please also tick the appropriate boxes below:

Which region are you from? Which group(s) do you participate in?

a) Northern Irrigation Region � a) Target 10 Statewide Executive �b) North East � b) Target 10 regional committee �c) South West � c) Target 10 working groups �d) Gippsland � d) Extension staff �e) Other / not relevant � e) Purchaser group �f) Other__________________________________________________________________ Are you? a) A farmer � d) An NRE employee �b) A University/TAFE provider � e) An agribusiness representative �c) A dairy factory representative �f) Other__________________________________________________________________

Thank you for completing this form. If I have not received your reply within 10 working days I will assume that you chose not to participate at this stage. The results of this survey will be made available to all stakeholder groups and committees as soon as possible.

306

Appendix 8 Facilitators guide for story selection process

The following guide details how the story selection process was facilitated during project meetings.

1. Write all the titles of the stories on the whiteboard, grouped by domain. Leave a space next to

each story for additional comments e.g.

Domain Title Comments 4 Using the consultant Strong, written by a farmer, incomplete, story not finished 4 Feeling empowered Moving story, beginning middle and end. Attribution to project is

questionable. Great story, not sure if it is about the project. 4 Better decisions for the

family Good solid story. Heard many times before. Small change yet crucial. Not sure about the dates mentioned

4 Now I understand OK, not enough information to really understand what it going on..

2. Ask the participants to read out all the stories belonging to the first domain of change. After

each story ask,

What is this story really about?

Write any comments next to the title on the white board as above.

3. When all the stories have been read out for the first domain, facilitate a discussion about the pros and cons of the stories and jot comments on the white board. Prompts can include:

• Is there any one story that particularly stands out to you?’ - why is that?

• Does a different story stand out to anyone else’?

• What about this story, no one has mentioned this?

• Are there any stories that stand out to you as being not so good - why is that?"

• What do you think of the stories in general?’

4. Ask the committee to move to a vote by hands, one vote per person. If there is total agreement,

about which story to select, move onto the next domain. If there is a spread of votes, facilitate a

further discussion and try to reach a consensus. In some cases, it may be better to choose two

stories for one domain, or choose non at all. A re-vote is sometimes helpful.

307

Appendix 9 Example vote form for Selection process

The following form was sent out to staff at the Central Executive Committee meetings, along with

the nominated stories from the four regions. The idea was that committee members would read the

stories before the meeting and record their impressions of each individual story in the form below –

to act as a memory jogger. The stories were then discussed during the meeting.

Domain: Title of story: Comments

Changes in farmer decision making skills

A) ‘More accurate decisions made now’ B) ‘Don’t need the consultant now’

C) ‘Becoming independent.

Changes in on-farm practice

G) ‘Production graphs help change practice’

H) ‘Top operator’

Changes in profitability

I) ‘I’ll not be milking cows when I am 55!’

J) ‘Doubled herd’s late lactation production’

Other type of changes:

D) ‘Information from neighbours’

E) ‘Meeting farmers needs’ F) ‘The most useful information ever received’