Monitoring and Evaluation of Community-Based Services for Orphans and Vulnerable Children in Soweto, Johannesburg by Edwin Tinashe Mutambanengwe 201336661 Submitted in fulfilment of the requirements for the degree Magister Artium in Social Work in the Department of Social Work of the Faculty of Humanities at the University of Johannesburg supervised by Prof Leila Patel Prof Adrian DuPlessis van Breda January 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Monitoring and Evaluation of Community-Based Services for
Orphans and Vulnerable Children in Soweto, Johannesburg
by
Edwin Tinashe Mutambanengwe
201336661
Submitted in fulfilment of the requirements for the degree
Magister Artium in Social Work
in the
Department of Social Work
of the
Faculty of Humanities
at the
University of Johannesburg
supervised by
Prof Leila Patel
Prof Adrian DuPlessis van Breda
January 2015
ii
Affidavit
This serves to confirm that I, Edwin Tinashe Mutambanengwe ID Number DN764606, Student
number 201336661 enrolled for the Qualification Master of Social Work in the Faculty of
Humanities herewith declare that my academic work is in line with the Plagiarism Policy of the
University of Johannesburg, with which I am familiar.
I further declare that the work presented in this dissertation is authentic and original unless clearly
indicated otherwise, and in such instances full reference to the source is provided. I do not presume
to receive any credit for such acknowledged quotations, and there is no copyright infringement in
my work. I declare that no unethical research practices were used or material gained through
dishonesty. I understand that plagiarism is a serious offence, and that should I contravene the
Plagiarism Policy, notwithstanding signing this affidavit, I may be found guilty of a serious criminal
offence (perjury). This would among other consequences compel the UJ to inform all other tertiary
institutions of the offence and to issue a corresponding certificate of reprehensible academic
conduct to whoever requests such a certificate from the institution.
Signed at Johannesburg on this 19th day of January 2015.
Signature: ______________________________
Print name: Edwin Tinashe Mutambanengwe
STAMP COMMISSIONER OF OATHS
Affidavit certified by a Commissioner of Oaths
This affidavit conforms with the requirements of the JUSTICES OF THE PEACE AND COMMISSIONERS OF OATHS ACT 16 OF 1963 and the applicable Regulations published in the GG GNR 1258 of 21 July 1972; GN 903 of 10 July 1998; GN 109 of 2 February 2001 as amended.
iii
Acknowledgements
By the Grace and Will of God.
This study would not have been possible if it was not for the following people, to whom I am
forever indebted:
Centre for Social Development in Africa (CSDA) – for availing and granting me this
opportunity to further my studies. I am forever indebted to the CSDA.
Prof Leila Patel – for epitomising excellence and inspiring such a remarkable quality in me.
You believed and invested in my potential since the inception of this study. Your approach
to supervision was graceful, insightful and it effectively inspired me to be more. Your
editorial prowess and attention to detail is mind-boggling. This study would not have been
possible if it was not for your proficiency and the guidance that you generously provided.
Prof Adrian DuPlessis van Breda – for providing strategic direction towards the completion
of this study. The dedication, concentration and level of organisation with which you
brought to this study is astounding. Your thought-provoking comments and insight always
brought a fresh perspective and provided sufficient impetus for completion of this study.
Maureen Coetzee – for being a bona fide social work cadre and continually providing
mentorship. You really exemplify organisational leadership and it’s a breath of fresh air
working under your headship.
Eunice Kutyauripo – for being the best mom in the world. You didn’t have to fully grasp
what I was studying to offer support and inspire confidence in me. All you knew was that it
was a worthy cause and that was good enough for you to encourage me to keep going.
Tshegofatso Keebine – for being a considerate partner; your encouragement, support and
patience helped translate this dream into reality.
Sipho Sibanda – for being a passionate and supportive friend and colleague. Your ability to
strike a balance between being industrious and being productive is awe-inspiring. Surely
success is a product of determination, diligence and reason.
Friends, colleagues and participants – for making this a success. Thank you for investing
your time and energy into this research even though there was no incentive to do so. I’m
forever grateful.
iv
Abstract
A lack of national policy and indicators to monitor and evaluate the implementation of the
developmental approach in the welfare field (specifically in child welfare) continues to be a
challenge (Patel, Grobbelaar, Selipsky, & Curtis, 2011). There are no appropriate guidelines and
procedures to evaluate new policy and approaches to child welfare practice, such as the community-
based approach to Orphans and Vulnerable Children (OVCs). Orphans constitute approximately
3.54 million of South Africa’s total population (Mathews, Jameison, Lake, & Smith, 2014, p. 92)
and there is an increase in the number of vulnerable children, that is, those living in circumstances
that deter holistic development (Fluke & Wulczyn, 2010; Mathews et al., 2014). These
circumstances, together with the paucity of empirical research on monitoring and evaluation (M&E)
to inform community-based services for OVCs informed this study.
Therefore, this study intended to evaluate the appropriateness and effectiveness of the M&E system
in a child welfare agency rendering community-based services for OVCs and to make
recommendations on how the system might be improved. In order to achieve this, a number of
objectives were linked to this goal. The first objective was to conduct a situation analysis of the
M&E system in a specific child welfare agency. Second, was to assess what works and what does
not work in the implementation of the agency’s design and the utilization of the agency’s data. The
third objective was to establish the utility and appropriateness of the existing M&E data for
organisational performance in rendering community-based services for OVCs. Finally, the study
endeavoured to make recommendations for an appropriate and effective M&E system for
community-based services for OVCs delivered in a child welfare setting.
A qualitative approach was employed and three data collection methods (a documentary study,
semi-structured interviews and a focus group discussion) were utilized sequentially. Data was
collected from agency M&E documents and participants who were social workers, supervisors and
managers within the agency. Thus, rich answers to descriptive and explanatory questions were
provided enabling the student to understand the complexity of the phenomenon that was being
studied. The study revealed that, first, there were no overall policies and guidelines to inform M&E
in the organisation; second, that the agency invented its own system as a stop gap measure, which
resulted in M&E not being prioritised. Third, the agency’s M&E system was inadequate due to a
lack of M&E budgets, technical expertise, knowledge and skills, and appropriate tools for M&E
data collection and analysis. The bureaucratic organisational structures limited the participation of
social workers and managers in shaping M&E processes. Furthermore, reporting mechanisms and
v
collaboration among stakeholders resulted in ineffective monitoring and evaluation of the new
directions in child welfare. The study concluded that there was a need for sound M&E systems
grounded in the developmental approach to child welfare, M&E theory and practice, and informed
by national policy, guidelines and indicators. Further recommendations for M&E policy, practice,
training of social workers and managers, M&E implementation at agency level, and for further
research were identified.
vi
Table of Contents
Affidavit .............................................................................................................................................. ii
Acknowledgements............................................................................................................................ iii
Abstract .............................................................................................................................................. iv
Table of Contents .............................................................................................................................. vi
List of Tables ..................................................................................................................................... xi
List of Acronyms .............................................................................................................................. xii
CHAPTER ONE: INTRODUCTION ............................................................................................ 13
1.1 Orientation to the Study......................................................................................................... 13
1.2 Motivation for the Study ........................................................................................................ 14
1.3 Research Questions, Goals and Objectives .......................................................................... 15 1.3.1 Research Question ...................................................................................................... 15 1.3.2 Research Goal ............................................................................................................. 15 1.3.3 Research Objectives ................................................................................................... 15
1.4 Overview of Research Methodology ..................................................................................... 16 1.4.1 Research Approach ..................................................................................................... 16 1.4.2 Research Site .............................................................................................................. 16
1.4.3 Research Design ......................................................................................................... 16 1.4.4 Population and Sampling ............................................................................................ 17
1.4.5 Data Collection ........................................................................................................... 18
1.4.6 Data Analysis and Interpretation ................................................................................ 18
1.4.7 Pilot Study .................................................................................................................. 19
1.5 Approach and Definition of Key Terms ............................................................................... 20 1.5.1 Child Welfare ............................................................................................................. 20 1.5.2 Orphan ........................................................................................................................ 20 1.5.3 Vulnerable Child ........................................................................................................ 20
1.5.4 Developmental Social Welfare ................................................................................... 21 1.5.5 Monitoring .................................................................................................................. 21
1.5.6 Evaluation ................................................................................................................... 21 1.5.7 Indicators .................................................................................................................... 21 1.5.8 Social Worker ............................................................................................................. 21
1.5.9 Community-Based Services ....................................................................................... 21
1.5.10 Orphans and Vulnerable Children (OVCs) ................................................................ 22
1.6 Limitations of the Study ......................................................................................................... 22
1.7 Ethical Considerations ........................................................................................................... 23
1.8 Chapter Outline ...................................................................................................................... 23
1.9 Conclusion ............................................................................................................................... 24
CHAPTER TWO: LITERATURE REVIEW ............................................................................... 25
2.1 Introduction ............................................................................................................................ 25
2.2 The Development of Child Welfare Services in South Africa ............................................ 26
vii
2.2.1 South Africa’s Constitution ........................................................................................ 27 2.2.2 Global and Regional Policies ..................................................................................... 27
2.2.3 National Policies and Frameworks ............................................................................. 28 2.2.3.1 The National Action Plan for Orphans and Vulnerable Children ................ 28 2.2.3.2 The Children’s Act No. 38 of 2005 .............................................................. 29
2.3 Approaches to Child Welfare Services ................................................................................. 31 2.3.1 The Child Protection Approach .................................................................................. 31
2.3.2 Family Services Approach ......................................................................................... 32 2.3.3 Community Care Model ............................................................................................. 33 2.3.4 The Child-Focused Model .......................................................................................... 34 2.3.5 South Africa’s Approach to Child Welfare Services ................................................. 35
2.4 The Developmental Paradigm ............................................................................................... 37 2.4.1 Developmental Social Welfare ................................................................................... 37 2.4.2 The Developmental Approach to Child Welfare ........................................................ 38
2.4.3 Approaches to Community-Based Services for OVCs .............................................. 40 2.4.3.1 The Isibindi Model ....................................................................................... 41 2.4.3.2 The Isolabantwana Model ............................................................................ 41 2.4.3.3 The Asibavikele Model ................................................................................ 42
2.4.4 Key Ideas on Community-Based Child Protection .................................................... 43
2.5 Monitoring and Evaluation of OVCs .................................................................................... 43
2.6 Approaches to Monitoring and Evaluation .......................................................................... 45 2.6.1 Defining Monitoring and Evaluation ......................................................................... 45 2.6.2 Rationale for Monitoring and Evaluation ................................................................... 46
2.6.3 Approaches to Monitoring and Evaluation ................................................................ 47 2.6.4 Evaluation Approaches Based on Scope .................................................................... 47
2.6.5 Evaluation Approaches Based on an Explicit Philosophy ......................................... 48
2.6.6 Participatory Monitoring and Evaluation (PM&E) .................................................... 50
2.6.7 Utilization-Focused Evaluation .................................................................................. 51
2.7 Conceptual Framework of the Study .................................................................................... 53
2.8 Conclusion ............................................................................................................................... 55
CHAPTER THREE: RESEARCH DESIGN AND METHODOLOGY .................................... 56
3.1 Introduction ............................................................................................................................ 56
3.2 Research Approach ................................................................................................................ 56
3.3 Type of Research .................................................................................................................... 56
3.4 Research Design and Methodology Overview...................................................................... 57
3.5 Phase 1: Documentary Study................................................................................................. 58 3.5.1 Population and Sampling ............................................................................................ 59
3.5.2 Data Collection Method ............................................................................................. 60 3.5.3 Data Collection and Analysis ..................................................................................... 60
3.6 Phase 2: Semi-structured Interviews .................................................................................... 61 3.6.1 Population and Sampling ............................................................................................ 61 3.6.2 Data Collection ........................................................................................................... 63 3.6.3 Pilot Study .................................................................................................................. 64 3.6.4 Data Analysis ............................................................................................................. 64
3.7 Phase 3: Focus Group Discussion.......................................................................................... 65
viii
3.7.1 Population and Sampling ............................................................................................ 66 3.7.2 Data Collection Method ............................................................................................. 66
3.7.3 Data Collection Tool .................................................................................................. 67 3.7.4 Data Analysis ............................................................................................................. 67
3.8 Strategies for Ensuring Trustworthiness ............................................................................. 68 3.8.1 Credibility ................................................................................................................... 68 3.8.2 Confirmability ............................................................................................................ 68
3.8.3 Transferability ............................................................................................................ 69 3.8.4 Member Checking ...................................................................................................... 69 3.8.5 Dependability ............................................................................................................. 69 3.8.6 Triangulation .............................................................................................................. 70
3.9 Reflexivity Statement ............................................................................................................. 70
3.10 Ethical Considerations ........................................................................................................... 72 3.10.1 Avoidance of Harm .................................................................................................... 73
3.10.2 Informed Consent ....................................................................................................... 73 3.10.3 Avoidance of Deception ............................................................................................. 74 3.10.4 Privacy, Confidentiality and Anonymity .................................................................... 74 3.10.5 Actions and Competence of the Student .................................................................... 75
3.10.6 Collaboration with Contributors ................................................................................. 75 3.10.7 No Compensation ....................................................................................................... 75
3.10.8 Publication of Findings .............................................................................................. 75
3.11 Conclusion ............................................................................................................................... 76
CHAPTER FOUR: PHASE ONE FINDINGS: DOCUMENTARY ANALYSIS ...................... 77
4.1 Introduction ............................................................................................................................ 77
4.2 Overview of the Agency’s Programme Structure ................................................................ 78
4.3 Results Emanating from the Documentary Study ............................................................... 80 4.3.1 ‘High’ Level Documents Governing M&E Processes ............................................... 81
4.3.2 ‘Low’ Level Components: Forms/Templates for Collecting Raw M&E Data .......... 86 4.3.3 Processes and Tools Used to Translate Data into M&E Indicators ........................... 90 4.3.4 Management of Processes .......................................................................................... 92
4.3.5 M&E Reports of Indicators ........................................................................................ 93
4.4 Conclusion ............................................................................................................................... 96
CHAPTER FIVE: PHASE TWO FINDINGS: M&E IN PRACTICE ....................................... 97
5.1 Introduction ............................................................................................................................ 97
5.2 Demographic Profile ............................................................................................................... 98
5.3 Overview of Findings Emanating from the Semi-Structured Interviews ....................... 100
5.4 Programme Structure of the Agency .................................................................................. 101
5.5 Knowledge of the Principles that Govern Community-Based Services for OVCs ......... 103 5.5.1 Understanding of the Meaning and Purpose of M&E .............................................. 103
5.5.2 Understanding of the Meaning and Utility of Indicators ......................................... 105
5.6 Community-Based Services for OVCs ................................................................................ 108
5.7 ‘High’ Level Documents ....................................................................................................... 109 5.7.1 Overall Policy on M&E ............................................................................................ 110
ix
5.7.2 M&E Guidelines ....................................................................................................... 111 5.7.3 Business Plans .......................................................................................................... 112
5.7.4 Indicators .................................................................................................................. 113
5.8 M&E Processes at the Point of Service Delivery Level ..................................................... 116 5.8.1 M&E Planning .......................................................................................................... 116 5.8.2 Social Workers’ Attitudes Towards M&E ............................................................... 118 5.8.3 Measurement of Community-Based Services for OVCs ......................................... 119
5.8.4 Statistical Information .............................................................................................. 120 5.8.5 Tools and Templates ................................................................................................ 122 5.8.6 Soweto Care System as the Agency’s M&E System ............................................... 124
5.9 M&E Data Processing, Analysis and Use ........................................................................... 126 5.9.1 Role of Supervisors in the M&E Process ................................................................. 127
5.9.2 Role of Managers in the M&E Process .................................................................... 128
5.10 M&E Reports ........................................................................................................................ 129
5.11 Training Needs ...................................................................................................................... 132
5.12 Recommendations for Improving M&E in the Child Welfare Field ............................... 133
5.13 Conclusion ............................................................................................................................. 134
CHAPTER SIX: PHASE THREE FINDINGS: PARTICIPANTS’ OVERALL
PERSPECTIVES AND RECOMMENDATIONS ...................................................................... 136
6.1 Introduction .......................................................................................................................... 136
6.2 Overall M&E Policy and Guidelines .................................................................................. 137
6.3 Organisational Structure ..................................................................................................... 138
6.4 Knowledge of the Principles Governing Community-Based Services for OVCs............ 140
6.5 Community-Based Services ................................................................................................. 141
6.6 M&E Tools and Processes in the Agency ........................................................................... 142
6.7 M&E Data Processing, Analysis and Use ........................................................................... 146
6.8 Reporting ............................................................................................................................... 148
6.9 Training ................................................................................................................................. 149
6.10 Recommendations ................................................................................................................. 150
6.11 Conclusion ............................................................................................................................. 152
CHAPTER SEVEN: SUMMARY OF FINDINGS, CONCLUSIONS AND
RECOMMENDATIONS ............................................................................................................... 153
7.1 Introduction .......................................................................................................................... 153
7.2 Summary of Key Findings ................................................................................................... 154 7.2.1 M&E System for OVCs in a Child Welfare Setting: Current Situation and Gaps ... 155
7.2.1.1 Overall M&E Policy and Guidelines ......................................................... 155 7.2.1.2 Organisational Structure ............................................................................. 156 7.2.1.3 Knowledge of the Principles Governing Community-Based Services for
OVCs .......................................................................................................... 156 7.2.1.4 Community-Based Services ....................................................................... 157
7.2.1.5 M&E Tools and Processes within the Agency ........................................... 157
x
7.2.1.6 M&E Data Processing, Analysis and Use .................................................. 158 7.2.1.7 Reporting .................................................................................................... 159
7.2.1.8 Training ...................................................................................................... 159 7.2.2 M&E System for OVCs in a Child Welfare Setting: Evaluation of the M&E Utility
and Appropriateness ................................................................................................. 160 7.2.2.1 Overall M&E Policy and Guidelines ......................................................... 160 7.2.2.2 Organisational Structure ............................................................................. 160
7.2.2.3 Knowledge of the Principles Governing Community-Based Services for
OVCs .......................................................................................................... 161 7.2.2.4 Community-Based Services ....................................................................... 161 7.2.2.5 M&E Tools and Processes in the Agency .................................................. 161 7.2.2.6 M&E Data Processing, Analysis and Use .................................................. 161
7.2.2.7 Reporting .................................................................................................... 162
7.3 Conclusions............................................................................................................................ 162
7.4 Recommendations for M&E Policy and Practice .............................................................. 165
7.5 Recommendations for M&E Implementation at Agency Level ....................................... 165
7.6 Recommendations for Further Research ........................................................................... 167
7.7 Final Word ............................................................................................................................ 167
References ....................................................................................................................................... 168
Annexures ....................................................................................................................................... 179 A. Data Components ......................................................................................................... 180
B. Agency M&E System Analysis .................................................................................... 182
C. Consent Form for Research Participants ....................................................................... 187
D. Semi-Structured Interview Schedule (Social Workers) ................................................ 189
E. Semi-Structured Interview Schedule (Supervisors) ...................................................... 193
F. Interview Schedule (Managers) .................................................................................... 197
G. Focus Group Discussion: Questioning Route ............................................................... 201
H. Flowchart of the Agency ............................................................................................... 202
I. Analysis of the Agency’s Community-Based Services for OVCs ................................ 203
J. Daily Diary .................................................................................................................... 206
K. Cluster and Sequence of Documents Analysed ............................................................ 207
L. Merged Caseload from Supervisors Collated at Managerial Level .............................. 208
M. Internal Statistics ........................................................................................................... 213
N. DSD Statistics ............................................................................................................... 225
O. Request for Permission Letter ....................................................................................... 230
P. Approval Letter ............................................................................................................. 232
xi
List of Tables
Table 2.1 Key Approaches, Ideas and Principles
Table 4.1 Description of Documents Studied
Table 5.1 Demographic Profile
Table 5.2 Work Experience Profile
Table 5.3 Overview of Semi-Structured Interview Findings
xii
List of Acronyms
ACRWC African Charter on the Rights and Welfare of the Child
AIDS Acquired Immune Deficiency Syndrome
CBD Central Business District
CP Care Plan
CPR Care Plan Review
CSDA Centre for Social Development in Africa
CSP Case Supervision Plan
CWSA Child Welfare South Africa
CYCC Child and Youth Care Centre
DCWA Developmental Child Welfare Approach
DSD Department of Social Development
DSW Developmental Social Welfare
HIV Human Immunodeficiency Virus
IDCW Integrated Developmental Child Welfare
IDP Individual Development Plan
M&E Monitoring and Evaluation
NAP National Action Plan
NGO Non-Governmental Organisation
NPO Non-Profit Organisation
NPAC National Plan of Action for Children
OVC Orphans and Vulnerable Children
PM&E Participatory Monitoring and Evaluation
RSP Reunification Services Plan
SAPS South African Police Services
SCS Soweto Care System
UN United Nations
UNCRC United Nations Convention on the Rights of the Child
UNICEF United Nations Children Fund
13
Chapter One: Introduction
1.1 Orientation to the Study
Developmental social welfare (DSW) is characterised as “an integrated and comprehensive
system of social services, facilities, programmes and social security to promote social
development, social justice and the social functioning of people” (White Paper for Social
Development, 1997, p. 5). It hypothesizes an all encompassing approach to poverty, which
entails unifying social and economic goals (Gray, as cited in Hölscher, 2008, p. 116). It is
distinguished internationally as a new approach to social welfare in the Global South and
particularly in South Africa where the approach has been adopted as formal government
policy (Gray, 2008; Midgley, 2001; Patel, 2005). However, there is a lack of an overall
national policy and indicators to monitor the implementation of the developmental approach
in the welfare field, which remains a major gap (Patel et al., 2011). The result is that there are
no standardized systems and procedures for monitoring the various types of child welfare
services that are mandated by the Children’s Act No. 38 of 2005 (hereafter referred to as the
Act).
The Act requires that child welfare agencies deliver integrated services that include
prevention services for children and their families, child protection against abuse and neglect,
early intervention, family preservation, rehabilitation and reintegration. Values of
empowerment, collaboration, respect for cultural and spiritual diversities and participation are
promoted. Emphasis is placed on building human capabilities and strengthening social
support systems in the family and community (Schmid, 2009). The Act also promotes best
practices, which entail being child-focused and outcome oriented in child welfare planning
and engagement (Schmid, 2009). A key principle informing the new approach to child
welfare practice is that services should be family-centred and community-based. This is a
new focus for child welfare agencies, whose services were previously residual, inadequate,
discriminatory disjointed and expert driven. It is a transition to a more organized,
participatory and democratic model characterised by respectful and inclusionary practices
(Schmid, 2013). Even though foster care has traditionally been delivered as a community-
based service, developmentally oriented community care for OVCs is relatively new. These
new directions in service delivery are not being monitored and evaluated. While community-
14
based care and support models are being implemented by child welfare agencies, little is
known about the effectiveness of these programmes and how to monitor and evaluate their
performance (Patel et al., 2011).
A Monitoring and Evaluation (M&E) system is therefore needed to improve the quality and
the scope of service delivery, inform planning, guide resource allocation and demonstrate
results which are important for purposes of accountability to key stakeholders (Noyoo, 2005;
World Bank, 2002). Moreover, M&E has a strategic role to play in informing policy making
processes, improving the relevance, responsiveness, efficiency and effectiveness of welfare
policy reforms and programmes (UNICEF, 2009b, p. 8).
There is limited literature on M&E of community-based services for OVCs. However, with
the introduction of the new policy directions and approaches to the alleviation of the OVC
crisis (such as the DSW, Integrated Developmental Child Welfare (IDCW), Children’s Act
No. 38 of 2005), the student wished to explore how M&E could be administered more
appropriately and effectively within the child welfare setting. A conceptual framework for
this study was devised that employs participatory and utilization focused management tools
relevant to evaluating current M&E systems in child welfare agencies in South Africa.
1.2 Motivation for the Study
This study was motivated, primarily, by professional interest and a realisation of the gap that
exists in the child welfare field. Having been employed in the child welfare sector since 2010,
the student has become an ardent professional in that field. Fouché and De Vos (2011)
support that an interest in a specific field often leads to the identification of gaps or defects in
the service delivery system, which in turn inspires research. The realisation that, currently,
there is a lack of proper M&E systems and specific indicators (Patel et al., 2011) in the child
welfare field also inspired this research. Child welfare agencies have had to improvise to
meet this knowledge gap. Unfortunately, these strategies remain inadequate to inform
agencies of the outcomes and impact of their services for OVCs.
The number of OVCs within the overall child population in South Africa continues to
escalate, primarily due to the AIDS pandemic (Mathews et al., 2014). Insight into this heart-
rending social condition also stirred this study (Bless & Higson-Smith, 2006) as there is need
15
for child welfare agencies to come up with M&E mechanisms, including indicators, to
evaluate OVC intervention policies, and have effective management tools to ascertain
whether community-based services for OVCs are really bringing about change.
Finally, the student was inspired to conduct this study in order to enrich the knowledge base
of the social work profession (Fouché & De Vos, 2011). Findings and recommendations of
this study will provide practical explanations, resolutions and robust ideas on the current
M&E situation in the child welfare field and make recommendations for future action.
1.3 Research Questions, Goals and Objectives
With reference to the information presented thus far, the student formulated the research
question, goal and objectives. These provide guidance for the rest of this study.
1.3.1 Research Question
The guiding question for this study is: What monitoring and evaluation system exists in child
welfare agencies rendering community-based services to OVCs and how might the M&E
system be improved?
1.3.2 Research Goal
The goal of the research is to evaluate the appropriateness and effectiveness of the M&E
system in a child welfare agency rendering community-based services for OVCs and to make
recommendations on how the system might be improved.
1.3.3 Research Objectives
The objectives of this study are to:
1. Conduct a situation analysis of the M&E system in a specific child welfare agency.
2. Assess what works and what does not work in the implementation of the agency’s
design and the utilization of the agency’s data.
16
3. Establish the utility and appropriateness of the existing M&E data for organisational
performance in rendering community-based services for OVCs.
4. Make recommendations for an appropriate and effective M&E system for community-
based services for OVCs delivered in a child welfare setting.
1.4 Overview of Research Methodology
1.4.1 Research Approach
A research approach is referred to as the rationale that is necessary in the management of a
research endeavour and common sense needed to inform strategy devised for that research
(Leedy, 2010). A qualitative approach was utilised for this study as it aimed to obtain
participants’ account of a situation (Fouché & Delport, 2011). This approach was also
consistent with the qualitative methods that were employed in this study as described by
Neuman (2000). Since the study intended to understand the participants’ experiences of the
M&E system in the agency, obtain their perceptions and the meanings and attitudes attached
to it, this approach was thus appropriate for the purpose of the study which is discussed
further in Chapter 3.
1.4.2 Research Site
The research site for this study was a specific child welfare agency, whose name has been
concealed for ethical reasons. That specific child welfare agency was used as a case study to
explore the M&E system largely within the child welfare field. The agency employs about 70
social workers. It has over 100 years of experience in the child welfare field and renders
services in the greater Johannesburg area. The study was limited to community-based
services for OVCs in Soweto rendered mainly through the agency’s Foster Care Unit (foster
care services for children and their families in formal care).
1.4.3 Research Design
Fouché, Delport, and De Vos (2011) define research design as the overall planning of any
scientific research leading to the accomplishment of the objectives. A research design guides
a researcher in the collection, analysis and interpretation of information obtained. This study
17
was therefore exploratory and descriptive in nature in view of the fact that the phenomenon
which was being studied was complicated. The student explored and described the M&E
system at a specific child welfare agency in terms of the ‘as is’ situation and the ‘what could
be’ situation. Following that, recommendations were generated of systemic improvements in
the M&E system that could be more appropriate and effective within a child welfare setting.
This was consistent with an applied research study, which this study specifically was, as it
delved into practical ways of addressing M&E issues (Neuman, 2000).
1.4.4 Population and Sampling
Neuman (2000) characterizes a population as the total set from which individuals
encompassing a study are selected. For data collection in the first phase (documentary study),
the population was defined as all agency documents that were related to M&E. A purposive
sampling method was used to collect a manageable number of documents selected on the
basis that they should be relevant to the research questions (refer to Annexure B).
There were three populations in the second phase of data collection that consisted of semi-
structured interviews. First, was a population of social workers which was defined as all the
social workers working in the Foster Care Unit rendering services to children and families in
Soweto. The population constituted 20 social workers. Four of these social workers were
selected to participate in the study using a purposive sampling method and criteria explained
in Chapter 3, Section 3.5.1. Second, was a population of supervisors, which was defined as
all unit supervisors who render supervision services to social workers, and are involved in the
collection and the collation of statistics. Two supervisors were purposively chosen to take
part in the study using the selection criteria described in Section 3.5.1 in the third chapter.
The population consisted of eight supervisors. Managers made up the third population. This
population consisted of four managers who oversee M&E functions and are involved in the
compilation of ‘high’ level documents (i.e. annual business plans, progress reports) and
collation of agency statistics. Using a purposive sampling technique, two managers were
selected. The conditions listed in the relevant section above were decisive factors in the
selection of the managers.
18
The third data collection phase (focus group discussion) was held with the four social
workers, two supervisors and two managers, all of whom had been interviewed by the student
in Phase 2. The agency’s advocacy officer was purposively selected and also incorporated in
this discussion on the basis that she uses statistics for lobbying and influencing policy and
that she contributed to the selection of appropriate documents for Phase 1. In total, the focus
group discussion comprised nine participants.
1.4.5 Data Collection
The collection of data for this study was divided into three separate phases which, however,
complemented each other. Data collection was closely linked to the attainment of the
objectives specified in this study.
The initial phase (documentary study) was conducted in fulfilment of research
objectives one and two. This entailed, first, locating agency data relevant to M&E and
determining that which could be analysed. Second, was an analysis of relevant M&E
data to describe the current agency M&E system, which might provide suggestions
for the ‘what could be’ situation.
The second phase was conducted with the view to fulfil research objectives two and
three and it consisted of semi-structured interviews. This phase constituted interviews,
first, with social workers, second, with supervisors, and then with managers. Overall,
this phase investigated the implementers’ opinions of what could make up an
appropriate and effective M&E system.
The third phase of data collection (focus group discussion) was conducted to obtain
participants’ perceptions of the findings of the study, and to explore options for
improving the agency’s M&E system. This discussion was made up of social workers,
supervisors and managers who had been interviewed in the second phase of data
collection. The agency’s advocacy officer was also incorporated in this discussion.
1.4.6 Data Analysis and Interpretation
The analysis and interpretation of data is a process that involves examining, sorting,
categorizing, comparing and synthesizing data (Neuman, 2000). Schurink, Fouché, and De
19
Vos (2011) describe data analysis as a process through which codes and categories are
formed of collected data. Ultimately, codes and categories are put together to form themes.
Data analysis for this study was consistent with the process described above. In the
documentary analysis phase, a case study database was created to manage and group the
agency M&E documents according to type and level at which they are produced. A relevant
data analysis tool developed by the student (refer to Annexure B) was then used to describe
the agency M&E situation. Data categories, which were coherent with each other and related
to a specific level in the M&E system of the agency, were created. Data was then coded using
the open code system and was further interpreted seeking possible explanations for collected
data and linkages among them. Appropriate visuals were subsequently developed to present
the data.
Data collected through semi-structured interviews and the focus group discussion was
analysed in the manner suggested by Creswell (2009, p. 126-129). It suggests how data ought
to be managed; the processes related to reading and writing memos; description of data,
drawing of patterns, coming up with themes and coding; and representation of data. An
exhaustive description of these processes is outlined in Chapter 3 of this study.
1.4.7 Pilot Study
It is crucial for newly constructed data collection instruments to be pilot tested thoroughly
before utilization in the main investigation (Strydom & Delport, 2011). The data collection
instruments for this study were thus piloted to eradicate elements of ambiguity and lack of
clarity so that accurate information required to meet the research objectives could be
obtained. The data collection tool for the documentary study was piloted to ascertain whether
it would collect relevant M&E documents. The semi-structured interview schedules were
piloted with two participants selected using the same sampling method as that of this study.
However, these participants did not participate in the main study.
20
1.5 Approach and Definition of Key Terms
A set of conceptual distinctions and ideas were adopted in order for the goal and objectives of
this study to be accomplished. The study was informed by a theory based approach combined
with participatory M&E principles and ideas, and it views M&E as an important management
tool to improve organisational performance. Table 2.1 provides an illustration of the key
approaches, ideas and principles informing the study and how they were organised for the
achievement of the research purpose.
The following are key terms and concepts that have been defined as they are significant to
this study.
1.5.1 Child Welfare
Child welfare is “the traditional term [used] for the network of policies and programs
designed to empower families, promote a healthy environment, protect children and meet
children’s needs” (Kirst-Ashman, 2007, p. 243).
1.5.2 Orphan
The term ‘orphan’ is used in the study to refer to a child, person below the age of 18 years,
who has lost their mother (maternal orphan), father (paternal orphan), or both biological
parents (double orphan) through death (Children’s Act No. 38 of 2005).
1.5.3 Vulnerable Child
A vulnerable child is “one who is living in circumstances with high risks and whose
prospects for continued growth and development are seriously threatened” (Fluke &
Wulczyn, 2010, p. 64). In South Africa, vulnerable children include those who are poor;
disabled; destitute or abandoned; displaced or terminally ill; living with a terminally ill parent
or guardian; born of a teenage or single mother; and living with a parent or an adult who
lacks income-generating opportunities (Smart, 2003).
21
1.5.4 Developmental Social Welfare
Developmental social welfare is referred to as “an integrated and comprehensive system of
social services, facilities, programmes and social security to promote social development,
social justice and the social functioning of people” (White Paper for Social Development,
1997, p. 5)
1.5.5 Monitoring
Mark, Greene and Shaw (2006, p. 6) define monitoring as “a systematic and continuous
collection and analysis of information about the progress of a programme over time.”
1.5.6 Evaluation
Evaluation is a process whose aim is to establish the outcome of the programme in relation to
its stated objectives and expected impact (Rabie & Cloete, 2011).
1.5.7 Indicators
Indicators refer to “specific, concrete, quantitative or qualitative variables that provide a valid
and reliable way to measure achievement, assess performance or reflect changes connected to
an activity, project or programme” (UNAIDS, 2010, p. 77).
1.5.8 Social Worker
According to the Children’s Act No. 38 of 2005, is any person who is registered or deemed to
be registered as a social worker in terms of the Social Service Professionals Act No. 11 of
1979.
1.5.9 Community-Based Services
This term entails “the provision of comprehensive high quality services in the home and
community in order to promote, restore and maintain people’s maximal level of comfort,
social functioning and health in the least restrictive setting possible” (NAP 2009-2012, p. 9).
22
1.5.10 Orphans and Vulnerable Children (OVCs)
The Children’s Act No. 38 of 2005 defines a child as any person under the age of 18 years.
On the other hand, the Department of Social Development makes provision for a person to
remain under the foster care system until they attain the age of 21 years. Thus, a vulnerable
child is described as any person under the age of 21 years who runs the risk of being abused,
abandoned, neglected, infected or affected by HIV/AIDS (Thiele, 2005). The term OVCs
does not refer to children infected and affected by HIV and AIDS only; it also includes
children who are orphaned and vulnerable.
1.6 Limitations of the Study
The following have been identified as the limitations to this study:
Even though a substantial amount of agency M&E documents were obtained by the
student, there were some documents that were inaccessible due to agency policies
which the student surmises could have provided more clarity.
The study focused, mainly, on the agency’s Foster Care Unit and used only nine staff
members of the agency delivering services in a particular area. This sample may not
be considered to be representative of the general population.
The study focused on social workers, supervisors and managers directly involved in
conducting M&E duties. However, no clients were interviewed, which could have
given the research another perspective on how the M&E system could be and
resonated well with the suggested participatory nature of M&E. Since children
(OVCs) and their foster parents are primary recipients of community-based services,
their voices and contributions should be included in future research endeavours aimed
at advancing knowledge production in this field.
A qualitative approach was employed as this resonated well with the descriptive
nature of the study. This study could however have drawn a much clearer perspective
of the ‘what could be’ agency’s M&E situation and generated additional
recommendations if quantitative information had been obtained and analysed. For
instance, an analysis of the agency’s current statistical information could have
23
provided clarity on what data the agency should actually collect, analyse and produce.
This should be the subject of further research.
1.7 Ethical Considerations
Ethics refer to a set of agreed upon moral principles that provide rules, which govern
expected behaviour and conduct with participants (Strydom, 2011a). In conducting this study,
the student conformed to a set of ethical principles that are widely accepted. Written consent
was obtained from the management of the agency where the research was conducted before
soliciting support from the staff to participate in the study. Accurate and complete
information was provided for participants to fully understand the study and make a voluntary
and reasoned decision about their participation (Taylor, 2000). The participants were assured
that the study was not an evaluation of their individual performance or the agency, but of the
suitability of their M&E system for other OVC programmes. Issues of confidentiality were
emphasised appropriately and the student applied all means to protect the privacy of the
agency documents and participants. Recording instruments were used with the knowledge
and consent of the participants
1.8 Chapter Outline
The study is divided into seven chapters and is presented as follows:
Chapter 1: It includes the introduction and orientation to the study as well as definition of key
terms pertinent to the study. It also gives a general orientation to the research methodology
employed.
Chapter 2: This is a literature review and conceptual framework. It also provides the context
of the study.
Chapter 3: Deals with the methodology of this study as the study in itself comprises various
innovations in the use of existing methodologies aimed at achieving the set goal and
objectives.
24
Chapter 4: This is a presentation of the Phase 1 data collection findings, i.e. findings
emanating from the documentary analysis.
Chapter 5: This deals with the presentation of findings emanating from the semi-structured
interviews conducted as the second phase of data collection.
Chapter 6: This presents findings emanating from the focus group discussion, which was the
last phase of data collection.
Chapter 7: This chapter sums up the study by providing key findings, conclusions and
recommendations.
1.9 Conclusion
This chapter serves as a general orientation to the study. The following chapter will provide
the literature review and conceptual framework of the study guiding the research, which is a
synthesis of the theory based approach, participatory M&E, and M&E as a management tool
(utilization focused approach). The following chapter will also discuss the theories, principles
and concepts contributing to the theoretical framework.
25
Chapter Two: Literature Review
2.1 Introduction
South Africa’s welfare system was re-organised from an inequitable social treatment
approach to a right-based developmental approach (Patel, 2005). This newly adopted
developmental approach posits a new paradigm for welfare services. It recognizes the need
for integrated family-centred and community-based services, employs a generalist and
strengths-based approach to service delivery, and advocates community development and
developmental welfare services (Patel, 2005, p. 156). This approach is rights-based and is
based on the Constitution and the Bill of Rights which guarantee the rights of children to
services and to meeting their basic needs. The Constitution also spells out the rights of
children to legal and socio-economic services; protection against maltreatment, neglect, abuse
or degradation; family and parental care and to alternative care when removed from the
family environment (Constitution of South Africa No. 108 of 1996). These ideas enunciated
above informed the White Paper for Social Welfare in 1997 which advocates a
developmental approach to child welfare including child protection, which is discussed
below.
A new Children’s Act No. 38 of 2005 was adopted to give effect to the policy and legislation.
Child welfare agencies currently deliver services according to this legislation. As indicated in
Chapter 1, there is no agreed national monitoring and evaluation system to determine what
progress has been made with the implementation of the legislation and the extent to which
services have been transformed. In addition, there are no guidelines to evaluate the impact of
the services in protecting and promoting child well-being.
In this chapter, I shall provide a brief background in part one to the development of child
welfare in South Africa with particular reference to the current policy and legislative
environment. Thereafter, in part two, I shall review the approaches to child protection
services internationally with a view to understanding how international thinking and past
welfare policies shaped South Africa’s child welfare services for OVCs. In part three, the
developmental approach to welfare will be outlined with reference to the basic tenets of the
developmental approach to child protection. Since the focus of the study is on M&E of
26
community-based services for OVCs, parts four and five will provide an overview of the
approaches to M&E and particularly those that are pertinent to this study. The conceptual
framework of the study is based on a synthesis of the theoretical approaches, which will be
outlined in Section 2.7.
2.2 The Development of Child Welfare Services in South Africa
The development of child welfare services in South Africa was primarily shaped by colonial
and apartheid forces. Schmid (2013) acknowledges that these forces influenced the structures,
legislation, policies and priorities in current child welfare practice. During the pre-colonial
era, traditional principles of collaboration and collectivism influenced the role of families and
communities in providing care for the most vulnerable children and families (Loffell, 2000).
However, colonialism and apartheid ushered in a new socio-economic system that
marginalized the black masses (Patel, Schmid, & Hochfeld, 2012, p. 214). Segregation co-
existed with a deficiency of services to the majority. By and large, the welfare system was
disjointed and prejudiced towards white people in urban areas (Schmid, 2013). Being residual
in nature and expert driven, this racially based welfare system did little to address the plight
of the poor, a situation which triggered violence and opposition to the status quo (Patel, 2005;
Schmid, 2013). As political parties strived to forge the way forward, it became evident that a
democratic South Africa, marked by transformation of frameworks and policies for child
welfare service delivery, was required (Schmid, 2013). Political initiatives began to highlight
the plight of children and brought children’s rights into the limelight. The context for
improving legislature, forming organisations and professionalizing child welfare services
began to form and a framework which promoted equality, acknowledged human rights and
directed wide participation of all racial groups, was adopted (Schmid, 2013).
In order to give effect to the new approach in the delivery of child welfare services, South
Africa adopted and endorsed various policies and laws. These policies and legislations
provide the mandate, outline the principles and guidelines for South Africa to render child
welfare services. The following discussion will illustrate various international and local
policies and legislation which South Africa adopted.
27
2.2.1 South Africa’s Constitution
The South African Constitution of 1996 (Act No. 108 of 1996) is a culmination of the
compromises that different political parties made to address past injustices (Patel, 2005). It is
representative of the character and course adopted by a democratic South Africa. UNICEF
(2009a) contends that the Constitution, the primary legislative framework, is unique in that it
grants children in South Africa specific rights that are aligned with international instruments
such as the United Nations Convention on the Rights of the Child (UNCRC) and the African
Charter on the Rights and Welfare of the Child (ACRWC). Moreover, the principles of
impartiality, free will, diversity, non-prejudice, social justice and participation are promoted
within the Constitution (Patel, 2005). Section 28 of the South African Constitution and the
Bill of Rights is instrumental in defining a child and it asserts children’s rights to identity,
culture, family or parental care, basic needs, socio-economic services, and protection from
maltreatment, neglect, abuse or degradation. The right to freedom, the right to legal
representation, and the right to holistic development (physical, mental, social, moral and
spiritual) are also enunciated in this section. The Constitution upholds the child’s best
interests as being paramount in every matter concerning the child. Overall, the Constitution
provides a national blueprint of a society that respects the equality and dignity of children and
adults alike (Patel, 2005). It safeguards socio-economic and cultural rights, as well as the
civil rights and freedoms of children (and adults), further demonstrating its compliance with
the international and regional treaties it endorsed (UNICEF, 2009c).
2.2.2 Global and Regional Policies
According to Patel (2005, p. 92), “the Constitution followed the lead of international human
rights instruments” such as the Universal Declaration of Human Rights (1948), the United
Nations Convention on the Rights of the Child (UNCRC, 1989), the Convention on the
Elimination of All Forms of Racial Discrimination Against Women (1979), among others.
Smart (2003) also identifies the World Declaration on the Survival, Protection, and
Development of Children (1990), the Millennium Summit (2000), and the United Nations
Special Session on Children (United Nations General Assembly, 2002) as key policy
frameworks that respond to the plight of children and are aligned with the South African
Constitution. Dutschke (2007), however, contends that the UNCRC and the ACRWC are the
most crucial legal instruments relevant in guiding programmes for all children, addressing
28
children’s rights and responding to the plight of OVCs. South Africa ratified the UNCRC on
16 June 1995 as a response to the needs of children and the disparities in child welfare
provision. By endorsing the UNCRC, South Africa reinforced its commitment to promote
children’s rights, protect children from abuse and neglect, encourage child development and
participation, prioritize children’s best interests and uphold the principle of equality (Smart,
2003).
On 7 January 2000, South Africa also accepted the main instrument of the African human
rights system, the ACRWC. The ACRWC is significant in addressing specific realities of
African children and it covers the whole spectrum of civil, political, economic, social and
cultural rights, placing particular emphasis on fairness and children’s contribution in the
decision making process. Through this policy framework, South Africa committed itself to
addressing harmful social and cultural practices and conceptualized the responsibilities and
duties of its communities towards children. Furthermore, South Africa devoted itself to
effectively enforce children’s rights and strengthen monitoring mechanisms, emphasizing the
role of parental and family care, as well as defining the duties and responsibilities of the child
towards the family and the community (Dutschke, 2007).
2.2.3 National Policies and Frameworks
2.2.3.1 The National Action Plan for Orphans and Vulnerable Children
The South African government formulated policies to materialize and harmonize the
aforementioned global and regional commitments into domestic law and to suit the new
developmental paradigm (Smart, 2003). The policies were then translated into strategies,
which are guided by goals and targets. The National Action Plan for Orphans and Vulnerable
Children (NAP, 2006-2008) is a key example of a framework with a greater focus on
coordination and monitoring of OVCs. It outlines a broader framework for the protection and
provision of comprehensive and integrated developmental services for OVCs as contained in
six key strategic areas and programmatic interventions (Department of Women, Children and
People with Disabilities, 2012a).
The first strategy of the NAP (2006-2008) emphasizes developmental interventions that
strengthen and support the capacity of families to protect and care for OVCs, such as
29
psychosocial support, food security and skills training. The second strategy relates to
mobilizing and strengthening community-based responses for the care, support and protection
of OVCs. It entails promoting community capacity building, involvement, coordination,
supporting good practice models and maintaining a database of services for OVCs at local
levels. The third strategy aims at ensuring that comprehensive legislation, policy, strategies
and programmes are in place to protect OVCs through collaboration, accessibility and
coordination of various role players involved in the delivery of services for OVCs. The fourth
strategy advocates a rights-based approach to service delivery. Service delivery mechanisms
should be developed and strengthened to provide essential services for OVCs as a right of
children. The fifth strategy aims to increase awareness and create a supportive environment
for OVCs through advocacy and the establishment of a comprehensive stakeholder
communication platform. The last strategy is inclined towards engaging civil society and the
business community to play an active role in supporting the plight of OVCs in line with
Article 16 of the UNCRC on state obligations regarding the impact of the business sector on
children’s rights.
The NAP (2006-2008) has a developmental agenda in which the plight of OVCs is prioritized
and family-oriented and community-oriented mechanisms are utilized to reduce vulnerability.
It presents an integrated and holistic approach to the protection and provision of services to
OVCs. Following a review process, the NAP (2009-2012) followed and continued to provide
guidance to government departments and sectors of civil society, building on work done in
the NAP (2006-2008). To date, South Africa takes pride in The National Plan of Action for
Children (2012-2017), which fuses existing international and national priorities into one
coherent framework. It provides, within the broader context of the South African Human
Rights Framework, a children’s rights impetus to national planning, implementation,
monitoring and evaluation of national priorities (Department of Women, Children and People
with Disabilities, 2012b). Its main principles are in harmony with those ascribed in the
national Constitution, UNCRC, ACRWC and the developmental approach to child welfare
services.
2.2.3.2 The Children’s Act No. 38 of 2005
As discussed earlier, there are various international conventions that inform policies relating
to services for children. The Children’s Act No. 38 of 2005 gives effect to South Africa’s
30
obligations concerning child well-being in accordance with the binding international
instruments (Department of Social Development, 2010). The Children’s Act is a child-
focused legislation that regulates the protection of and services for children in South Africa.
It gives the mandate for child protection agencies to render integrated services for children
(OVCs included) and their families in a developmental fashion. The fundamental tenet
underlying the Children’s Act is that these services should revolve around the family and the
community. The objectives of the Children’s Act are in accord with the strategies enunciated
in the NAP (2006-2008 and 2009-2012) and NPAC (2012-2017) which speak to holistic child
development. Its principles are also in agreement with those of the international and local
policies discussed above with emphasis, however, on “the best interests of the child” and on
community-based interventions (The Children’s Act No. 38 of 2005).
The discussion above regarding South Africa’s endorsement of international conventions and
the subsequent formulation of national policies and programmes for children shows a shift
from the old unjust child welfare system and the social treatment model to a new child
welfare approach which is unbiased, developmental and rights-based (Patel, 2005). It entails
a rearrangement of priorities in child welfare services whereby family-centred and
community-based care and access to rights by children take precedence over other forms of
care and services for children. Moreover, it presents ideas, values, policies and practices that
incline child welfare service delivery towards developmentalism, and accentuates the role of
monitoring and evaluation as a strategy for enhancing community-based interventions for
children (OVCs included). The Department of Social Development (2010) highlights that
through the policy frameworks and legislations discussed above, child welfare needs to lean
more to strengths-based, rights-based, people-oriented, empowerment and growth-focused
approaches. Success of this child welfare orientation depends upon the collaborative
partnerships of various role players.
Having discussed the policies and legislations that comprise the transformation of child
welfare service delivery in South Africa, it is also of paramount importance to consider
various international approaches to child welfare services. The view is to understand how
child welfare services being delivered on the international scene influence and shape South
Africa’s child welfare services for children.
31
2.3 Approaches to Child Welfare Services
Freymond and Cameron (2006) suggest that there are three types of child welfare responses,
namely the child protection, family services, and community care approaches. Gilbert, Parton
and Skivenes (2011) concur with this proposition but highlight that due to the complexity of
current approaches to child maltreatment, different countries are refocusing child welfare
practice and these models seem to exist in combination with each other. Moreover, Gilbert et
al. (2011) recognize an emerging strong interest in the well-being of children being displayed
by the media, policy and practice. Welfare services for children have undergone widespread
restructuring of professional services as well as in the cultural landscape, which necessitates
an acknowledgement of the different contexts of childhood (Ansell, 2005). According to
Gilbert et al. (2011), this is a reflection of societies turning towards an alternative child-
focused approach. The following section will discuss each of the four approaches that inform
child welfare provisions internationally. There is growing international interest in South
Africa’s developmental approach to child welfare.
There is a dearth of literature relevant to child welfare systems. Resultantly, the student relied
primarily on the work of Gilbert et al. (2011) and Schmid (2012) since they are the principal
authorities on the subject. Gilbert et al. (2011) present recent work on various approaches to
child welfare services from an international perspective. Schmid (2012) provides literature on
comparative child welfare systems and combines various approaches to child welfare
formulating a child welfare model relevant to South Africa.
2.3.1 The Child Protection Approach
Gilbert et al. (2011) identify England’s child welfare system (among other Anglophone
countries) as being characteristic of the child protection model. The child protection model is
reactive rather than proactive as it responds when children have already been harmed or are
on the verge of being harmed (Schmid, 2012). The primary responsibility for child care is, in
this model, assigned to parents who may be held responsible for being neglectful and abusive
towards children. The state functions as “watchdog” to ensure child safety (Gilbert et al.,
2011, p. 255) and social workers derive their authority from the courts to intervene and
render corrective parenting procedures (Schmid, 2012). In general, the system is separate
32
from other institutions (education, health and welfare) consequently creating a single-point
access. The child protection model has been criticised for overemphasizing “protection rather
than the overall welfare of the child” (Gilbert et al., 2011, p. 255). In the case of England,
such overemphasis resulted in “investigation becoming an end in itself” (Gilbert et al., 2011,
p. 62). Apart from this, the child protection model has also been criticised for being intrusive,
punitive, moralistic, individualistic, legalistic, and creating an adversarial relationship
between the state and the parent (Burman, 2003). Its focus on the vulnerable child overlooks
the other relations that exist between the child and its parents or broader family. This state of
affairs has the potential of creating divisions between the needs of the child and of parents or
broader family (Burman, 2003). Schmid (2012) criticizes the model for overly stressing
administrative and statutory procedures, and using out-of-home placements instead of
rendering primarily preventative and supportive services. According to Patel (2005), the child
protection model as an approach to child welfare services cannot effectively address the
needs of poor children and their families since it favours tertiary and expert driven responses.
Due to numerous criticisms levelled against the child protection model, several improvements
had to be made to address the bias of the system, increase family partnership activities,
promote interagency and interdisciplinary collaboration, and offer more positive and holistic
mechanisms for resolving disputes (Freymond & Cameron, 2006). Even so, other critics still
argue that this system must be fundamentally transformed as it fails children and families.
Others advocate the family services model, which is discussed below.
2.3.2 Family Services Approach
The family services approach epitomizes the work being done with children and families in
many European countries (Freymond & Cameron, 2006). Gilbert et al. (2011) identify
Sweden and France as classic examples of countries inclined towards the family services
model. In this model, the state shares the responsibility for child rearing with parents and
plays a supportive role seeking to strengthen family relations. Partnership between the state
and parents is promoted with solidarity being the core cultural value underpinning this model
(Schmid, 2012). The family services approach acknowledges that all families are prone to
encounter difficulties, hence they need support, such as parenting support, day care and youth
centres (Waldegrave, 2006). Universal, primary prevention is therefore offered to families as
support before they come into crisis (Schmid, 2012). Gilbert et al. (2011) contend that the
33
model implicitly recognizes that structural factors may cause child abuse, as much as the
psychological, marital and socio-economic problems of parents. In the case of Sweden, the
current Social Service Act (2001:453) clearly outlines the role of the local social service
authority in collaborating with families when rendering ongoing supervision and protection
services (Gilbert et al., 2011). Emphasis is placed on joint problem-solving and upholding the
best interests of the child especially when considering out-of-home care (Waldegrave, 2006).
Foster care is used in exceptional cases, with parents maintaining contact with their child, and
adoption is uncommon. According to Gilbert et al. (2011), the family services approach is a
therapeutic family-needs assessment oriented system. It has three main functions in Sweden
which relate to reporting a case, investigation of the reported case, and adopting either
voluntary (family service) or compulsory (child protection) care. Intervention is therefore less
punitive and allows for multiple access points (Schmid, 2012). Stigma is reduced by
integrating child protection activities into broader welfare activities (Freymond & Cameron,
2006) and children’s courts play a peace-making role (Gilbert et al., 2011).
The family services approach is, however, criticized for being too sensitive to parental needs
and overshadowing the needs of children (Schmid, 2012). It is also criticized for reflecting
dominant gender relations and adopting an individualist position considering its focus on the
nuclear family and individual family members (Schmid, 2012, p. 202). Like the child
protection model, it is strongly criticized for using middle-class, Western values to define
social constructs such as “childhood” and “child rearing” which makes it a rather limited
approach when dealing with migrant populations (Andersson, 2003). In an effort to find child
welfare responses that address the criticisms of the family services model and better reflect
community values and traditions, the community care model emerged. A discussion of this
model ensues below.
2.3.3 Community Care Model
Schmid (2012) explains that the quest to understand the relationship between parenting
capacity and structural factors led to the emergence of the community care model. It is
characteristic of countries such as New Zealand, Australia, the United States and Canada
where intervention is needed to address both structural and familial issues without
disregarding the cultural perspectives of minority and poor people (Freymond & Cameron,
2006). In this model, children are perceived to be naturally connected to the immediate and
34
extended family circle and culture (Burman, 2003). As such, collaborative decision making
processes are preferred and all family members are included and allowed to participate in
decision making. This model stresses the need to put in place measures to enhance the
participation of children and women. Final decisions regarding intervention are made based
on the family group’s knowledge of their situation, not on “expert” input (Schmid, 2012, p.
202). Emphasis in this model is placed on planning the fate of the child who has been or is
likely to be abused. It is inclined towards generating a comprehensive understanding of
children’s needs (Freymond & Cameron, 2006). Unlike the child protection model, the
community care model does not compromise the practices and norms of minority
communities. However, it is not a popular model given that it has surfaced when child
protection models are dominant. It conflicts heavily with the child protection model and is
criticized for being time consuming and limited in rendering preventative services (Schmid,
2012).
2.3.4 The Child-Focused Model
This is a child-centric perspective whose major component is that children are individuals
with “opinions, interests and viewpoints” (Gilbert et al., 2011, p. 170). Children are also seen
as members of their generational group, and society should recognize them. According to
Gilbert et al. (2011, p. 171), a way of conceptualizing this model is to distinguish between
three aspects in terms of: “children’s legal rights and organisational procedures; adults’
recognition of children as individuals with particular interests and needs in interactions with
adults; and the use of the individual child’s viewpoint as a way of interpreting what the world
means to children.”
The emergence of a child-centric orientation is reflected among countries such as Norway,
Germany and Denmark, which have ratified the UNCRC, promote child participation in
decision making and recognize the child as the main person in a child welfare case (Gilbert et
al., 2011). Within this model, children have an independent relation with the state. Focus is
not limited to harm and abuse, but on a child’s overall development and well-being. The
driver for intervention in this model is the individual child’s needs in a present and future
perspective, and society’s need for healthy and contributory citizens (Gilbert et al., 2011).
Should there be concerns about a child’s development, the state intervenes by offering
support and providing a wide range of early intervention and preventative services. It can also
35
adopt a more authoritative “defamilialization” role, where it serves as a parent but seeks to
“refamilialize” the child through foster care, kinship care or adoption (Gilbert et al., 2011, p.
255). Interventions using the child-centric model are adjusted and cognizant of children’s
needs, competencies and maturity. In other words, situations are viewed from the perspective
of the child and practitioners are trained to consider the child’s perspectives, as exemplified
by Belgium and Germany’s child welfare systems (Gilbert et al., 2011). The child-focus
model is a future oriented approach that considers the state as making a “social investment”
whilst upholding the principle of “individualization” (Gilbert et al., 2011, p. 255). This social
investment approach to child welfare is accompanied by behaviour regulation of parents in
particular, professionals and children themselves so that everyone fulfils their responsibility.
According to Gilbert et al. (2011, p. 253), the state plays a proactive and pre-emptive role,
whilst the family is obliged to promote their child’s best interest and develop skills and
competencies. Through working methods such as interviews and children’s groups, children
are encouraged to participate and express their views in decision making.
Whilst this approach has resulted in increased attention to children and has enhanced the
quality of their childhood, it is heavily criticized for challenging the traditional family-
centred approach. Gilbert et al. (2011) criticize it for placing children’s rights above parents’
rights and emphasizing parents’ obligations as caregivers (Gilbert et al., 2011). Moreover, the
paternalistic role played by the state is rather punitive and ruinous to families.
Defamilialization is not conciliatory or aligned with the family-centred approach. Instead, it
creates a substitutive and adversarial relationship between the state and parents (Gilbert et al.,
2011). In addition, this model is criticized for borrowing largely from the child protection and
family service models, putting emphasis on the child protection model. The child-focused
model is also criticized for providing services to promote child well-being, but in turn
demanding change and results on behalf of the child (Gilbert et al., 2011). Child-centred
methods are considered difficult to apply equally across agencies, as they seem to flourish
more in NGOs than other sectors (Gilbert et al., 2011).
2.3.5 South Africa’s Approach to Child Welfare Services
The discussion above of the models of child welfare services helps us appreciate how
international thinking influenced South Africa’s approach to child welfare services. As
alluded to earlier, colonialism and apartheid played a central role in shaping child welfare
36
services in South Africa (Loffell, 2000; Patel, 2005; Schmid, 2012). Through colonization,
the Anglophone child protection model was introduced to South Africa as the initial mode of
intervention (Patel, 2005). The advent of apartheid led South Africa into adopting Western
models which were deficit-based, treatment-oriented, residual and individualized (Dutschke,
2007). To date, child welfare services in South Africa largely reflect Anglo-American
systems (Patel, 2005; Schmid, 2009) and the “child protection model continues to hold sway”
(Schmid, 2012, p. 204). This is partly attributed to the influences of globalization and
international policies on children’s rights (Schmid, 2012). Loffell (2008) substantiates that
the child protection model remains relevant to the South African context, considering the
daunting challenges that children are confronted with in the society in which they live.
Schmid (2012) concurs with Loffell (2008) that poor access to health, education and welfare
services by children rationalize the adoption of the child protection model. The significance
of high rates of rural-urban migration, unemployment and violence, and the “poverty-AIDS-
crime complex” in increasing child vulnerability in South Africa is also acknowledged
(Schmid, 2012, p. 205). Nonetheless, Schmid (2012) argues that South Africa has entirely
orientated itself towards child protection, which presents unwieldy challenges in trying to
respond to the needs of vulnerable children. The Anglo-American child protection model
remains inadequate regardless of the different values, principles and intervention approaches
(such as tailor-made responses to child protection, family preservation and family group
conferencing) that are encouraged in it (Schmid, 2012). Patel (2005) asserts that the current
child protection approach in South Africa is costly. It allows the state to easily withdraw from
direct intervention by employing intermediate service agencies such as NPOs to deliver
services on its behalf. Many challenges have been noted about the under-funding of NPOs.
Confronted with the socio-economic challenges in South Africa, and the difficulties
associated with being orientated towards the Anglo-American child protection approach,
South Africa’s child welfare system requires an alternative approach (Schmid, 2012). Schmid
(2012, p. 200) argues that South Africa needs a more integrated, developmental and tailored
approach that combines protection issues with other extensive issues related to child well-
being. An integrated developmental child welfare model is thus advocated as being relevant
in the context of South Africa’s children and families (Patel, 2005; Schmid, 2012).
37
The next section will explore the developmental child welfare approach. It will first discuss
the themes, principles and thrusts underlying the developmental social welfare paradigm. An
outline of community-based service models and how they resonate with the developmental
approach will ensue, drawing specific examples from the South African context.
2.4 The Developmental Paradigm
2.4.1 Developmental Social Welfare
Since the developmental child welfare model derives its principles, themes and practices
from developmental social welfare (henceforth, DSW), it is essential to first explore DSW in
order to appreciate the developmental child welfare model. DSW is an approach that was
adopted by South Africa in 1997 as a deliberate and fundamental change in welfare policy
(Patel et al., 2012) from the previous racial, paternalist (Hölscher, 2008) and a social
treatment model of service delivery. It is a modernized, inclusive and people-centred welfare
system regarded as suitable for meeting the needs of South African citizens (Lombard, 2007).
DSW is a strengths-based approach that promotes growth of individuals, groups and
communities (Department of Social Development, 2006b). The key principles of DSW can be
succinctly listed as participation, empowerment, equity, accessibility, appropriateness, self-
reliance, social integration, transparency, accountability, universal access and sustainability
(Department of Social Development, 2011). The aim is to build human capacities and
enhance the lives of individuals, families and communities. A better balance between
treatment, preventive, promotive and developmental initiatives is needed (Patel, 2005).
South Africa’s endeavour to move towards a developmental approach was characterised by
the adoption of the White Paper for Social Welfare in 1997 which sets the parameters for
DSW (Department of Social Development, 2006b; Patel et al., 2012). A rights-based
approach which entails achieving a minimum standard of living, social justice and equity is
the first key feature informing DSW. The policy sets a commitment for equal resource
redistribution and equal access by all but paying particular attention to disadvantaged groups
and previously neglected communities (Patel, 2005). The second premise is that economic
development should be harmonized with social investments (such as provision of housing,
education, welfare services) that help alleviate poverty and build human capabilities (Patel et
38
al., 2012). Third, the policy accentuates citizen participation and decision making regarding
the development and in the delivery of services (Patel et al., 2012). Welfare pluralism is
identified as the fourth core feature of DSW implying a collaborative partnership of the state
with voluntary, informal and commercial sectors for effective social development as well as
the family and the community (Patel, 2005). Bridging the micro-macro divide is recognized
as the last core feature of the approach. It entails an integrated, generalist and multi-method
approach to the provision of developmental welfare and social work services (Patel et al.,
2012).
2.4.2 The Developmental Approach to Child Welfare
The DSW approach described above provides a broad framework for the application and
operationalization of a developmental discourse in child welfare (Schmid, 2012). Patel (2005)
propounds eight major thrusts of services underlying the developmental approach to child
welfare. First, it should be distinguished that the family (which includes immediate relatives,
more distant relations and friends) is a crucial entity for the protection, survival and
development of children. Services should therefore become family-driven and cognizant that
child’s rights and best interests are intrinsically interlinked with the rights of those of the
family network (Schmid, 2012). Second, the state collaborates with the family providing
supportive services to ensure its survival and development. Interventions are directed at
strengthening families and communities to face hardships and to challenge structural issues
(Schmid, 2012). Third, services which include financial and material resources, basic needs
and supportive services, should be delivered from a multi-method approach. According to
Patel (2005), emphasis is placed on the use of less restrictive and more methods that are
family-centred, community-based, supportive and developmental. Patel (2005) outlines that
the fourth focus should be on balancing preventative, rehabilitative and developmental
intervention strategies. The need to redirect services so that they are accessible, relevant and
developmental is identified as the fifth thrust. Services with a higher reach and impact (Patel,
2005) such as prevention, early intervention, family- and community-based strategies are
emphasized. Patel (2005) asserts that the sixth thrust entails making social services and
intervention a mechanism for addressing socio-cultural aspects. To achieve this, intervention
should be gender and culture-sensitive, generalist (Patel, 2005) and receptive of the “fluidity
and boundary permeability within family constellations” (Schmid, 2012, p. 209). The seventh
thrust pinpoints the role of services in protecting, promoting and facilitating access to rights
39
by children and families. Finally, a collaborative approach among different welfare players
(state, NGOs and private sector) to mainstream integrated youth development services into
current developmental welfare interventions is advocated (Patel, 2005).
Schmid (2012) asserts that an Integrated Developmental Child Welfare (IDCW) approach is
relevant in the South African context since it is cognizant of the principles of DSW,
encompasses Patel’s (2005) thrusts, and integrates numerous features of the family service
and community care models. IDCW also coincides with South Africa’s cultural notion of
Ubuntu in that a child is viewed as inherently intertwined with the family and that the child’s
voice should be heard concurrent to that of the family group (Schmid, 2012). It is a
participatory, collective, rights-based, preventative, collaborative and supportive approach
(Schmid, 2012, p. 206). The recognition of the child’s connection with the family network
and broader community is central in this approach and is regarded as a mechanism for
sustainable and greater protection of the child (Schmid, 2012). The IDCW is aware of the
connections between prevention, early intervention and statutory services but it stresses the
importance of multilevel interventions that speak to the entire context of the child and family.
Within this approach, much weight is placed on primary prevention services but integration
of services, intersectoral collaboration and multiple service strategies are preferred as they are
cost effective and easy to accomplish. As in DSW, the IDCW model proposed by Schmid
(2012) recognizes socio-economic integration as a critical facet in alleviating poverty
achievable through social security mechanisms and connecting child welfare initiatives with
support for people’s livelihood strategies, education, skills development and employment
among others.
This discussion at hand of the developmental approach to child welfare services is crucial in
informing how community-based services for OVCs should be rendered. As a developmental
response to the realities of children and families, community-based services need to break
away from resource intensive, individually focused and problem oriented services towards
family- and community-based approaches to child protection and development. This is
regarded as critical and more effective in responding to the OVC situation (Schenk, 2009).
Additionally, the thrust of services, principles of practice, beliefs and ideas within the
community-based model should be directly aligned with the developmental approach to child
welfare. The following section will outline the community-based services approach and
40
explore several examples within the South African content where it is being employed to
alleviate the plight of OVCs.
2.4.3 Approaches to Community-Based Services for OVCs
As articulated in the first chapter, the concept of community-based services entails “the
provision of comprehensive high quality services in the home and community in order to
promote, restore and maintain people’s maximal level of comfort, social functioning and
health in the least restrictive setting possible” (NAP 2009-2012, p. 9). It is a strengths-based
system of care with a holistic perspective of strengthening the child, the family and
community as a whole (Child Welfare Information Gateway, 2013). Natural support systems
are viewed as assets and the principles of community engagement and commitment are
esteemed (Thurman et al., 2008). In order to protect and improve the well-being of OVCs,
Smart (2003) attests that the community-based service model has a set of developmental
activities which cover the following:
Prevention and early intervention services: awareness campaigns, conducting
assessments, child protection against abuse and neglect;
Family preservation and strengthening community mobilization: to increase the
capacity of communities to identify vulnerable children and to design, implement, and
monitor their own OVC support activities;
Fostering community-based care and support of OVCs;
Integrated services: integrating OVC support with home-based care, voluntary
counselling and testing, and mother-to-child transmission prevention activities;
Providing training and support: for individual counselling and psychosocial support;
Rehabilitation and reintegration: therapy, family reunification services;
Strategy development and partnership formation: to create or maintain household
resources and community safety nets;
Supporting interventions: to reduce institutionalization and abandonment of children;
and
Monitoring and evaluating of OVC programmes.
The above illustration of developmental activities within the community-based service model
is all encompassing depicting how and what community-based services should encompass.
41
This discussion remains incomplete if practical examples from the South African context are
not identified to demonstrate how developmental child welfare is being operationalized
through the community-based service model. The following discussion will therefore focus
on the major community-based models of child welfare services in South Africa. This will
help provide insight on how community-based services for OVCs are being rendered and
monitored in South Africa lessons of which are of great value for the purposes of this
research.
2.4.3.1 The Isibindi Model
The Isibindi model was conceived in 2001 in Kwa-Zulu Natal with the view to provide
community-based services for orphaned and vulnerable children in the context of HIV/AIDS.
Through establishing partnerships with other welfare players, the model aims to create safe
and caring communities amidst the complexities and challenges of high HIV/AIDS
prevalence rates, high unemployment rates, high poverty levels and poor access to social
services (Wilson, 2008). It recognizes the collaborative role of the state, NGOs, civil society
and municipalities in rendering services for children but distinguishes Child and Youth Care
Workers (CYCWs) and mentors as vital players in rendering comprehensive and effective
community-based services for OVCs (Wilson, 2008). Community members with limited
education are selected and trained in child and youth care, and then deployed into their
communities to provide community-based services for OVCs. This model asserts that within
the collaborative structure of social service provision, CYCWs have a pivotal role to play in
enhancing networks and championing the rights of OVCs (Wilson, 2008). CYCWs spend
much time with children, sharing and influencing their experiences on a moment-by-moment
basis. They use children’s daily routines, activities, experiences and living environment to
provide therapeutic care and developmental opportunities (Wilson, 2008, p. 412). Income
generating projects, foster care programmes, child protection programmes, development and
empowerment programmes are some other service delivery components within this model.
The Isibindi model has been adopted by government as a national pilot programme.
2.4.3.2 The Isolabantwana Model
Isolabantwana, meaning ‘Eye on the Children’, is yet another approach to community-based
services for OVCs initiated in 2003 by Cape Town Child Welfare as a response to the
42
increasing OVC phenomenon, specifically in the context of child abuse and neglect
(Reynecke, Steyn, & Rankin, 2007). Natural helpers in the community are identified and
trained by Child Welfare South Africa (CWSA) as 'eye volunteers' that safeguard children in
their communities (Cape Town Child Welfare, 2013, p. 1). They are trained to be alert for
abuse, family crises and to provide preventative and early intervention services for children
and families through events, talks and workshops. Other training areas include counselling,
statutory intervention, children’s rights, parenting skills, domestic violence, HIV/AIDS,
substance abuse and the role of stakeholders such as SAPS (Reynecke et al., 2007). ‘Eye
volunteers’ assist by removing abused, neglected and exploited children from danger and
placing them in short-term emergency safe houses with place of safety parents until social
workers can intervene. The Isolabantwana model aims to support social workers, promote
community participation and to create a sustainable awareness and responsibility within the
community to care for children (Reynecke et al., 2007). Families and communities are
empowered through training to prevent abuse and neglect, and children are empowered to
protect themselves through life skills training. Rehabilitation and family reintegration
programmes are facilitated by volunteers who also cooperate and network with the police,
schools, churches, local government and organisations to protect the rights of children
(Reynecke et al., 2007).
2.4.3.3 The Asibavikele Model
In the face of financial constraints and human resources shortages to address the ever
increasing incidences of OVCs and family crises, the Asibavikele/Let us protect them
community-based service programme was introduced (CWSA, 2013). It is a nationally driven
and nationally co-coordinated programme that aims to facilitate community-based care and
support for OVCs in the context of HIV/AIDS. The Asibavikele project focuses mainly on
disadvantaged communities that are underserviced and is implemented through the
comprehensive infrastructure and collective action of CWSA and its members (CWSA,
2013). Asibavikele is a child centred model which effects community involvement in the
identification and care of OVCs. It sensitizes communities to children’s rights and helps
establish foster care and safe homes within communities. Its main focus is prevention and
early intervention, and capacitating children, their caregivers as well as community
volunteers (CWSA, 2013).
43
2.4.4 Key Ideas on Community-Based Child Protection
The examples cited above reflect how community-based initiatives are striving to incorporate
developmental policies, practices, ideas and values into their efforts to address the challenges
faced by OVCs and to create viable economies and societies where OVCs reside (Schenk,
2009). A range of services including psychosocial support, healthcare services, statutory
services, training and development, linking clients with resource systems, economic support,
and community development are provided for through this model. Key ideas and principles
informing community-based child protection include community participation, community
responsibility over the welfare of its children, intersectoral collaboration allowing for
multilevel interventions, multiple service strategies that allows for a range of service delivery
methods, and innovative community strategies (Patel, 2005; Smart, 2003; Wilson, 2008).
UNICEF (2009c) highlights that community-based services for OVCs operate using an
inclusive principle and provide tailored training and local criteria to assess vulnerability and
come up with innovative solutions that help protect, promote and facilitate children’s access
to rights.
To date, none of these community-based programmes have been rigorously evaluated for
their impact. Evidence-based evaluations are critical to illustrate whether these models are
effective in practice (Patel, in press). It is of significant interest to this study to discover how
these community-based services for OVCs are monitored and evaluated to realise the impact
of the services. The following section will therefore explore different approaches to M&E
and consider the ones suitable for the South African context.
2.5 Monitoring and Evaluation of OVCs
There is a paucity of literature on the monitoring and evaluation of community-based services
for OVCs locally and internationally. However, the World Bank (2013) advocates
participatory monitoring and evaluation (PM&E) as an appropriate evaluation approach to
OVCs which reduces unintended leaks to other interest groups and ensures that the needs of
the OVCs are well understood and, therefore, served as intended. PM&E is regarded as a
strategy to build community buy-in, co-ownership for the intervention, and supporting project
sustainability. The World Bank (2013) asserts that sound M&E for OVCs should define
44
measurable input, output, outcome and impact indicators rooted in programme objectives.
These indicators should be based on, first, the OVC indicators that are already available and
regularly recorded, and second, indicators that can reasonably be recorded and monitored
locally in terms of cost, time and ease of collection. The aim is to make data collection cheap
and as simple as possible, by using what is already available and collecting only the
minimum amount of additional information needed to make project-related decisions.
Thurman et al. (2009) conducted an evaluation study of community education and
sensitization as an OVC care and support strategy. This research conducted in Kenya,
focused only on prevention and early intervention as one aspect of community-based services
for OVCs in the context of HIV/AIDS. The study however pinpoints the importance of using
mixed methods of collection of evaluation data (interviews, focus group discussions) whose
emphasis is on participation and joint learning of key stakeholders. The study draws
conclusions to the effect that interventions targeted towards OVCs and community
development have a high impact in creating community awareness and fostering community-
based care of OVCs (Thurman et al., 2009). Direct family services coupled with community
education and sensitization of OVCs can enhance child protection, and mobilization and
support of community-based responses is key in addressing the plight of OVCs (Thurman et
al., 2009).
Nyangara et al. (2009) also conducted an evaluation study in Tanzania of an OVC
programme which aims to strengthen community-based responses to meet the needs of OVCs
and provide psychosocial support to OVCs. The OVC programme dubbed ‘Mama Mkubwa’
mobilizes, trains and provides ongoing technical support to volunteers who are instrumental
in providing community-based psychosocial support and running kids clubs. The training for
these community counsellors covers topics pertaining to psychosocial support, community
counselling techniques, and guidance in the implementation of kids’ clubs. The study design
was a case study and it employed methods such as document review, in-depth interviews and
observations. The study utilized participation of OVC stakeholders and experts (local and
global) in the evaluation process and sought to ensure consensus regarding study objectives
and other expectations. Results of this study reflect that developmental community-based
activities such as home visits, counselling and kids clubs need to be strengthened as they have
a high impact on the well-being of OVCs (Nyangara et al., 2009).
45
Important M&E lessons can be also drawn from the aforementioned community-based
services for OVCs within the South African context. Wilson (2008) underlines programme
activities that serve as mechanisms for M&E. At staff level, documentation of activities,
client contacts and case proceedings, reports, plans, diaries of events and compilation of
quantitative and qualitative statistics serve as mechanisms for monitoring and evaluation.
Mechanisms for M&E at supervisor level include; supervision of files, staff supervision,
collation of statistics and mentorship, team meetings and report canalization. Mechanisms for
M&E at managerial level include compilation and analysis of statistics, drawing up business
plans, and compiling reports for submission to funders (Wilson, 2008).
As highlighted in Chapter 1, a major gap still remains in that there is no overall national
policy and indicators to monitor the implementation of the developmental approach in the
welfare field (Patel et al., 2011). There is no uniformity in the M&E systems and procedures
across different child welfare agencies. Rather, the child welfare field is characterised by
tailored solutions to bridge this gap.
2.6 Approaches to Monitoring and Evaluation
2.6.1 Defining Monitoring and Evaluation
Monitoring and evaluation (M&E) are two distinct yet complimentary sets of organisational
activities; they are related but not identical (World Bank, 2006). As defined in Chapter 1,
monitoring is “a systematic and continuous collection and analysis of information about the
progress of a programme over time” (Mark, Greene, & Shaw, 2006, p. 6). It comprises of
inputs, processes, outputs, outcomes and impacts. Evaluation, on the other hand, aims to
establish the outcome of the programme in relation to its stated objectives and expected
impact (Rabie & Cloete, 2011). It assesses the design, relevance, implementation, efficiency,
effectiveness, impact and sustainability of a programme and draws conclusions about the net
benefits of a programme through the verification of values and standards (Rossi, Lipsey, &
Freeman, 2004). In its totality, M&E looks at what one sets out to do, what was
accomplished, and how it was accomplished.
46
As highlighted above, inputs, processes, outputs, outcomes and impacts are the main
components of M&E. It is therefore crucial to define these elements at this juncture.
According to Imas and Rist (2009), inputs refer to all types of resources (such as funding,
staffing, buildings and equipment) that contribute to the production and delivery of outputs.
Inputs relate to what is used to do the work in a project or a programme. Processes refer to
the actions or activities that translate inputs into desired products (National Treasury, 2007).
In essence, activities describe what project or programme implementers do, such as
facilitating, supplying or marketing of services (Imas & Rist, 2009). The tangible and
measurable products or services delivered as a result of the activities is what is referred to as
outputs. According to the National Treasury (2007, p. 6) outcomes are “medium-term results
for specific beneficiaries that are the consequence of achieving specific outputs.” Outcomes
relate to what a project or programme wishes to achieve and thus clearly convey the strategic
goals and objectives (National Treasury, 2007). Lastly, impacts refer to the long term changes
that result from the achievement of specific outcomes. Examples of impacts include poverty
reduction and job creation (Rabie & Cloete, 2011).
2.6.2 Rationale for Monitoring and Evaluation
Rabie (2011) asserts that M&E has a central role to play in generating and providing
continuous information about a programme. Kusek & Rist (2004) highlight that monitoring
informs stakeholders where a programme is at any given time (and overtime) in comparison
with set targets and outcomes. In other words, it establishes baselines which are invaluable
for painting an ongoing picture of progress (Rabie, 2011). In the light of this, Lewis, Lewis,
Packard and Souflêe (2001) clarify that stakeholders can keep track of inputs, activities and
accomplishments and assess whether services are being rendered in accordance to the plan,
budget and time frame. Conversely, evaluation provides reasons why a programme is failing
to reach targets and outcomes (Kusek & Rist, 2004). Stakeholders can therefore check for
consistency between criteria and standards set during the planning phase against the results
and make the necessary remedies for the programme to get back on track (Lewis et al., 2001).
Since the main objective of human service programmes is achieving client change, M&E
information is also crucial in informing stakeholders about the quality and the scope of
service delivery (Noyoo, 2005). Additionally, M&E information can be used in planning,
decision making, guiding resource allocation, showing progress in the use of allocated funds
47
and demonstrating results (Noyoo, 2005; World Bank, 2004). Given that child welfare
practice is guided by policies and driven by donor and state funding, Noyoo (2005)
emphasizes that M&E has a strategic role to play in enhancing accountability to key
stakeholders and providing a more robust basis for raising funds and influencing policy.
Indeed M&E of community-based services for OVCs should influence policy making
processes, help improve the relevance, responsiveness, efficiency and effectiveness of policy
reforms and programmes (UNICEF, 2009b). Finally, M&E creates organisational memory
which is vital for future learning (Rabie, 2011).
2.6.3 Approaches to Monitoring and Evaluation
There are numerous approaches to M&E as shall be discussed below. However, Rabie and
Cloete (n.d.) underscore the importance of attaining a fit relating to the purpose of M&E,
fundamental values of evaluation and the methodology if the most rigorous results are to be
achieved. Rabie and Cloete (n.d.) systematically classify M&E approaches according to the
scope, philosophy and design of the evaluation. Based on these classification categories, the
following section discusses different approaches to M&E delineating their strengths and
shortcomings.
2.6.4 Evaluation Approaches Based on Scope
The scope of the evaluation entails whether evaluation will broadly encompass several or all
dimensions of a programme, or will be narrowly focused and limited to a specific aspect
(Rabie & Cloete, 2011). Comprehensive evaluations focus on more than one aspect of
evaluation whilst focused evaluations will consider one aspect which could be a policy,
programme, product, development sector, geographic area, intervention stage, or staff
performance. Rabie and Cloete (2011) cite the following as main approaches based on scope:
community-based evaluation; sectoral evaluations; geographical evaluations; policy
evaluation; programme and project evaluation; product evaluation; input evaluation; process
evaluation; output evaluation; outcome evaluation; impact evaluation; systemic evaluation;
integrated evaluation and meta-evaluation.
Stufflebeam and Shinkfield (2007) explain that these approaches are useful in producing
thorough M&E information but they require system-wide buy-in; highly skilled personnel to
48
run computer based M&E programmes and conduct large-scale statistical analysis; politically
effective leaders who are accountable and committed to use the results for improvement.
These approaches present clear evaluation goals, aim at improving service delivery and
promote innovation (Stufflebeam & Shinkfield, 2007). Nonetheless, approaches based on
scope are heavily criticized for being too focused and limited (Rabie & Cloete, 2011). They
are also criticized for relying heavily on quantitative information hence complex data analysis
is based on a limited scope of variables (Stufflebeam & Shinkfield, 2007). Moreover, these
approaches are regarded as politically volatile since they are used to identify responsibility
for successes and failures. This can create unhealthy competition, unrest and
misrepresentation of facts in order to ensure positive evaluation reports (Stufflebeam &
Shinkfield, 2007).
2.6.5 Evaluation Approaches Based on an Explicit Philosophy
Rabie and Cloete (2011) highlight that previous evaluation approach classifications attempted
to distinguish between quantitative (scientific) and qualitative (interpretative) approaches,
and value-driven versus use-driven evaluation approaches. They however dismiss these
classification attempts as imprecise. They therefore suggest a clearer distinction of evaluation
approaches in terms of the underpinning philosophy as being either participation driven or
theory driven. Participation-driven evaluation philosophies broadly aim to engender
“development, empowerment and to create a shared understanding of the programme
between the evaluators and the beneficiaries and decision makers” (Rabie & Cloete, 2011, p.
15). Evaluation is conducted from a team perspective in which the evaluator is the team
leader assisted by stakeholders in planning, conducting and analysing the evaluation. Main
principles surrounding this approach are democratic participation, shared responsibility and
collaboration (Rabie & Cloete, 2011). Examples of such approaches include evaluative
enquiry, appreciative inquiry, critical theory evaluation, democratic evaluation,
empowerment evaluation and utilisation focused evaluation.
Theory driven evaluation philosophies, on the other hand, have an inclination towards the
scientific approach and they aim to contribute towards knowledge expansion (Rabie &
Cloete, 2011). Such approaches include clarification evaluation; realist evaluation; cluster
evaluation; illuminative evaluation; and goal free evaluation. Theory-driven evaluation is
defined as “the systematic use of substantive knowledge about the phenomena under
49
investigation and scientific methods to improve, to produce knowledge and feedback about,
and to determine the merit, worth and significance of evaluands” (Donaldson & Lipsey, as
cited in Rabie & Cloete, 2011, p. 16). The main purpose of the theory-driven evaluation
approach is “to determine the extent to which the programme of interest is theoretically
sound, to understand why it is succeeding or failing, and to provide direction for programme
improvement” (Stufflebeam & Shinkfield, 2007, p. 37). Approaches in this category are
typified by a model of a programme’s logic normally in the form of a flowchart illustrating
how inputs are processed to produce the required outcomes (Stufflebeam & Shinkfield,
2007). Rabie and Cloete (2011, p. 16) explain further that these approaches allow for
mapping and design of complex programmes not simple linear cause-and-effect relationships.
Furthermore, there seems to be a “theory of change” linking evaluation with intended
improvements in practice.
Since the thrust of this research is to understand the current M&E system in a specific child
welfare agency with the view to provide direction for improvements towards an appropriate
and effective M&E system for community-based services for OVCs, the theory driven
evaluation philosophy is of much relevance. This philosophy allows for the student to
systematically observe events, analyze material and conduct a network analysis in order to
conceptualize a context dependent theory or flowchart that explains how a programme works
(Stufflebeam & Shinkfield, 2007).
Theoretical approaches that this study might evaluate include family centred and community-
based services; generalist services; rights of children with regards to implementing statutory
mandates; prevention, early intervention, rehabilitation and promotion of child wellbeing;
holistic approach to client intervention (multidisciplinary approach); linking clients with
resource systems and partnerships; empowerment of client groups; capacity building and
participation in collaborative forums.
In order to offset the disadvantages and criticisms levelled against the evaluation studies
based on scope, and to incorporate principles of democratic participation, responsiveness and
collaboration in M&E, the student reckons participatory monitoring and evaluation, and
utilization-focused evaluation as relevant M&E approaches for this study. Participatory
monitoring and evaluation, and utilization-focused evaluation approaches allow for a mixed
50
method design to be employed as they are dynamic and situational. The following section
shall then discuss these approaches to M&E as relevant for the purposes of this study.
2.6.6 Participatory Monitoring and Evaluation (PM&E)
PM&E is part of a wider historical process (Rabie & Cloete, 2011) regarded by the World
Bank (2004) as an effective approach that ensures collection and availability of useful
information through participation. PM&E draws on various participatory research methods.
King (as cited in Mathison, 2005, p. 291) advocates participatory monitoring and evaluation
(PM&E) as “a relevant and overarching approach that transcends the limited traditional
methods by actively involving both programme staff and participants in decision making in
activities related to the planning and implementation of programmes”. According to Rabie
and Cloete (2011), PM&E fosters development and empowerment, creates shared
understanding of the programme between the evaluators, beneficiaries and decision-makers,
and enhances programme ownership. Imas and Rist (2009) describe it as an adaptive, action-
oriented strategy which builds stakeholder capacity and facilitates collaboration and shared
decision making for increased utilization of evaluation results. It is also significant in
building accountability and transparency and taking corrective actions to improve
performance and outcomes (World Bank, 2013).
PM&E is relevant for this study since it is developmental and illuminates on the principles of
participation, inclusion, diversity, shared responsibility, empowerment and democracy
(World Bank, 2004). It allows organisations seeking to develop M&E systems to align its
programmes and policies to the human rights based approach. King (as cited in Mathison,
2005) explains that the key feature to this philosophy is the substantial role of participants in
making decisions about the evaluation process not only providing information. The World
Bank (2004) explicates that PM&E involves stakeholders at different levels working together
to identify problems, collect and analyze information, and generate recommendations.
PM&E is a useful approach that could be adopted by the agency where this study is being
conducted as it involves the participants in the process of evaluation and accesses their
knowledge and experience of what is feasible (Mathison, 2005). It is responsive to
stakeholder concerns and combines the M&E process with participants’ values, needs and
standards, thereby generating relevant, high quality data. Moreover, PM&E enhances
51
confidence and capacity in the process of development. It is a non-judgmental, educational
and growth-focused intervention (Noyoo, 2005, p. 232). PM&E establishes partnerships and
local ownership of projects which enhances local learning, management capacity, and skills
development. Resultantly, timely and reliable information for management decision-making
is provided (World Bank, 2004).
PM&E is, however, often criticised for being insufficient and less objective - this is the
reason why not much weight is attached to it (Greene, 2006, p. 129). It is a time-consuming
process that has potential for domination and misuse by some stakeholders to further their
own interests. Nevertheless, PM&E is of relevance to this study since the purpose is to learn
from what is being done, to build on the actual experiences of staff, and to generate a sense of
ownership among them of a system that would be most useful to them. Guijt and Gaventa
(1998) also argue PM&E provides an opportunity for development agencies in welfare fields,
such as child welfare, to focus better on their ultimate goal of improving lives and reducing
vulnerability. Moreover, it (PM&E) is compatible with the participatory community-based
service approach and helps sustain local initiatives by ensuring effective use of community
resources (Guijt & Gaventa, 1998).
2.6.7 Utilization-Focused Evaluation
The utilization-focused approach to evaluation is also appropriate for this study since it
enhances shared understanding of the evaluation process among stakeholders, helps bring
support, promotes participation, and strengthens organisational capacity (Patton, 2004).
Information generated using this approach is useful in informing users of what needs to be
done to fulfil their objectives. Patton (2004, p. 277) asserts that since evaluation cannot be
value-free, utilization-focused evaluation will help frame the evaluation according to the
values of the users of evaluation findings. It prides in putting the client group in the driver’s
seat. In this approach, representative stakeholders clarify the outcomes, indicators,
performance targets, data collection plan and intended use of the findings (Patton, 2004).
Stufflebeam (1999, p. 60) describes utilization-focused evaluation as “highly personalistic,
dynamic and situational”. It is a user oriented approach that can be tailored to meet any
programme evaluation assignment allowing the facilitator to play different roles (negotiator,
trainer, measurement expert, internal colleague, external expert, analyst, spokesperson or
mediator) depending on circumstances. Utilization-focused evaluation allows for different
52
evaluation types, designs, focuses and purposes to be created, and any type of data to be
generated depending on utility (Patton, 2004, p. 283). It addresses action-oriented questions
and utilizes techniques that the client group identifies with which builds confidence of
stakeholders in the evaluation process. Generally, utilization-focused evaluation is “active-
reactive, adaptive and situationally responsive,” emphasizing a fit between the client group’s
needs and contextual dynamics (Stufflebeam, 1999, p. 62). Throughout the evaluation
process, the evaluator balances the concern for utility with provisions for validity and cost-
effectiveness.
Even though the utilization-focused approach is praised for being ubiquitous and pragmatic
(Patton, 2004), geared for impact (Stufflebeam, 1999) and allowing use of multiple methods
(Alkin, 1995), it has its fair share of shortcomings. Stufflebeam and Shinkfield (2007),
highlight that the efficacy of this method is downplayed by the need for a substantial outlay
of time and resources. It is criticized for lacking credibility since it works closely with
intended users. Similar to the participatory approach, it is argued that users are given too
much control which makes the approach vulnerable to corruption by stakeholders with
conflict of interest (Patton, 2004). According to Stufflebeam (2004), it is a gruelling task to
get a representative user group that concurs on time, ethics, data collection methods and
reporting issues. Moreover, effective implementation of this approach requires a highly
competent, confident and flexible evaluator who will not compromise professional standards.
The turnover of involved users typical of this approach can upset the evaluation process
(Patton, 2004).
Despite these criticisms highlighted above, the benefits of using the utilization-focused
evaluation approach far outweigh the losses of using it. Developmental organisations, such as
child welfare agencies, wishing to develop M&E systems will find value in adopting and
adapting the utilization-focused approach as it allows for mixed methods designs
(Stufflebeam and Shinkfield, 2007). PM&E and Patton’s utilization-focused evaluation are of
particular relevance to this study since they are multi-method approaches that may be adopted
and modified by the agency where this study is being conducted. PM&E and Patton’s
utilization-focused evaluation approach will facilitate the agency in adopting a M&E system
that incorporates input, output, outcome, process-centred, impact and systemic M&E methods
(Kusek & Rist, 2004; Rabie & Cloete, 2011; Rossi et al., 2004) into its current community-
based programme for OVCs. Since the research site is a child welfare agency, prone to
53
transformations caused by internal and external socio-political environments and a
combination of diversities within and across settings, mixing M&E approaches will aid in
increasing rigor and providing in-depth understanding of the relationship between contextual
factors and outcomes (Lawrence et al., 2013, p. 6).
Lawrence et al. (2013) highlight that due to the nature of challenges faced by child welfare
agencies (high staff turnover, bureaucratic pressures, low worker compliance with evaluation,
and high caseloads), social workers will comply more with evaluation tasks that are
participatory and emphasize the utility of the evaluation process. Therefore, child welfare
agencies should adopt and implement an M&E design which is participatory in nature,
utilization-focused and strives to link M&E findings with positive change (Lawrence et al.,
2013) if staff are to be invested in the evaluation. Moreover, the M&E design should
encourage knowledge building, facilitate partnership formation, promote flexibility and
pragmatism, facilitate change of roles and redefinition of success, enhance collaboration and
create feedback loops that return the M&E findings back to the field for improvement
purposes (Lawrence et al., 2013). For these reasons, participatory monitoring and evaluation
and utilization-focused M&E are of great value to this study hence the rationale for choosing
them. The following section will highlight the conceptual framework of the study which will
lead to the conclusion.
2.7 Conceptual Framework of the Study
This section highlights the approach that the study is taking and presents the manner in which
ideas and conceptualizations are organized in order to achieve the research purpose. Since the
purpose of the study is to evaluate how the developmental approach is being implemented in
the delivery of community-based services for OVCs, the M&E approaches are outlined in
part 5 above. The M&E approach informing the study is, first, a theory based one. This
theory based M&E approach highlights the key ideas and principles directing how services
should be delivered (approaches, levels, types and methods) and for what purposes. Second,
this theory based approach will be combined with participatory M&E principles as it involves
working with practitioners and managers in the agency engaged in M&E. Even though no
data will be collected from children or their families, this study acknowledges the value of
beneficiary participation in M&E through their interaction with practitioners and managers in
54
information gathering and decision making, which is the thrust of participatory M&E.
Finally, the utilization-focused approach is considered useful and vital to this study since it
develops shared understanding of the evaluation process among stakeholders, encourages
mutual support and participation, and bolsters organisational capacity (Patton, 2004). Given
that the study seeks to ensure that evaluation will form part of a continuous programme
improvement process in the agency, utilization focused evaluation is key to this study
because it generates information useful in informing users on what to do to fulfil their
objectives.
In order to accomplish the research purpose and achieve the research objectives, this study
has a set of conceptual distinctions and ideas. As highlighted above, the study is informed by
a theory based approach combined with participatory M&E principles and ideas, which forms
broader part of an overall organisational management tool, which is the utility aspect. The
following table provides an illustration of the key approaches, ideas and principles informing
the study and how they are organized in order to achieve the research purpose.
Table 2.1 Key Approaches, Ideas and Principles
Approach Key Ideas / Principles
Theory based approach - Family centred and community-based services
- Generalist services (individuals, groups, families and communities)
- Empowerment of client groups
- Capacity building
- Rights of children in regards to implementing statutory mandates
- Prevention, early intervention, rehabilitation and promotion of child
wellbeing
- Linking clients with resource systems and partnerships
- Holistic approach to client intervention (multidisciplinary approach)
- Participatory and collaborative strategies undergirding service
delivery
(Rabie & Cloete, 2011; Stufflebeam & Shinkfield, 2007)
Participatory M&E - Involvement of key stakeholders (professionals, managers,
paraprofessional staff) in the M&E process
- Joint decision making and joint learning of stakeholders at various
levels
- development and empowerment of users (stakeholder capacity
building)
- programme ownership and shared understanding of the evaluation
programme
- alignment of policies and programmes with the human rights
approach
- mutual respect
- catalyzes commitment to taking corrective actions
(Imas & Rist, 2009; Mathison, 2005; Noyoo, 2005; Rabie & Cloete, 2011)
55
Approach Key Ideas / Principles
M&E as part of an
organisational management tool
(informed by the Utilization-
focused approach)
- M&E results are used to design programmes and intervention
strategies
- M&E generates information on what needs to be done to fulfil
objectives
- M&E is used to unify stakeholders by promoting participation and
support
- M&E strengthens organisational capacity to deal with challenges and
provides information on how to deliver effective services to the
beneficiaries
- Information generated from M&E is used for planning and
implementation purposes
- Helps stakeholders come up with implementation strategies
- M&E processes are used to generate useful results that will be fed
back into objectives, goals, plans, and strategies
(Stufflebeam & Shinkfield, 2007; Rabie & Cloete, 2011)
2.8 Conclusion
In this chapter, I have traced the development of child welfare services in South Africa and
considered how this influenced current policies, legislation and approaches to child welfare.
International approaches to child welfare services have also been considered outlining how
they influenced the South African approach to child welfare services. This led to a discussion
on the developmental approach in social welfare broadly and how it resonates with child
welfare services, particularly those that are community-based. Since the focus is on M&E of
community-based services for OVCs, the subsequent sections explored the various
classifications of M&E approaches but placed emphasis on those the student considered
relevant and appropriate for this study, namely participatory M&E and utilization-focused
evaluation. Lastly, the student developed the conceptual framework of this study. In the
following chapter, the research design and methodology will be discussed.
56
Chapter Three: Research Design and Methodology
3.1 Introduction
The focus of this chapter is on the empirical study that was undertaken to evaluate the
appropriateness and effectiveness of the M&E system in a child welfare agency rendering
community-based services for OVCs in Soweto, Johannesburg. The student intended to
answer the following question: What monitoring and evaluation system will be most
appropriate for organisations rendering community-based services to OVCs? In this chapter,
the research approach, design, methodology, ethical aspects related to the study and
procedures used to achieve the research objectives will be discussed.
3.2 Research Approach
Since the study endeavoured to develop an understanding of complex phenomena and
intended to answer descriptive and explanatory questions, it employed a qualitative approach
in which three qualitative data collection methods (documentary study, semi-structured
interviews and focus group discussion) were employed sequentially. By so doing, the student
managed to capture data in the first phase of data collection (documentary study), which
helped paint an overall picture of the agency’s M&E system. Qualitative data gathered from
the second and the third phases of data collection (semi-structured interviews and focus group
discussion respectively) allowed participants to give much richer answers to the questions
posed by the student, thereby providing very useful insights into the phenomena that the
student was studying (Burns & Grove, 2009, p. 51). Moreover, the student was able to use the
qualitative data gathered in the second and third phases of data collection to complement data
gathered from first phase (documentary study), that is, to fill in the gaps that were not
addressed in the first phase of data collection and getting the lived experiences of the M&E
system from the implementers.
3.3 Type of Research
This research was exploratory and descriptive in nature. More specifically, it was an applied
research study, since it sought to make recommendations for an appropriate and effective
57
M&E system for community-based services for OVCs delivered in a child welfare setting.
Applied research studies often seek to find solutions to policy issues and to assist
practitioners in the social service field with practical ways of addressing issues (Neuman,
2000, p. 24).
3.4 Research Design and Methodology Overview
An overview of the study design and methodology will be discussed in this section.
Subsequent sections will illuminate on specific data collection methods employed for each of
the phases.
The case study design was utilised with the view to thoroughly explore the M&E of
community-based services for OVCs in Soweto, Johannesburg at a well-established child
welfare agency. The agency was used as a case study of M&E broadly in the child welfare
sector, as it has an M&E system of some sort, including statistical data and indicators. The
student did not study the whole agency but purposively selected the Foster Care Unit, since it
renders the majority of specific community-based services for OVCs in Soweto.
Given the situation that it was not clearly understood how the devised indicators and statistics
generated by the agency for internal and external purposes informed the overall M&E system
of the agency, a case study design was the most appropriate research design to employ. The
fact that the M&E system of this agency was not well understood or documented posed
intricacies in conducting this study, which could be best overcome by utilizing the case study
research design (Yin, 2004). Through the case study design, the student managed to elicit
very useful information by asking “how” and “why” questions, which gave more clarity and
enabled the student to understand the contextual conditions relevant to the agency’s M&E
system (Yin, 2004, p. 3). Moreover, the case study design allowed the student to use different
data collection methods to elicit comprehensive information (Yin, 2004, p. 5).
Data collection was divided into three phases:
The first phase, entailing a documentary study, was conducted in fulfilment of
research objectives one and two. In order for this phase to achieve its purpose in
fulfilling the first two objectives, it was divided into two distinct yet complementary
58
sub-phases, 1a and 1b. Sub-phase 1a involved locating agency documents relevant to
M&E and determining that which could be analysed. Sub-phase 1b entailed the
analysis of relevant M&E documents to describe the ‘as is’ situation and help suggest
the ‘what could be’ situation.
The second phase, which consisted of semi-structured interviews, was conducted with
the view to answer research objectives two and three. At this stage, the student
conducted an enquiry of the implementers’ opinions of what could constitute an
appropriate and effective M&E system. This phase was divided into three sub-phases:
sub-phase 2a constituted of interviews with social workers; sub-phase 2b was
composed of interviews with supervisors; and sub-phase 2c constituted of interviews
with managers.
During the third phase of data collection, a focus group discussion was held with the
social workers, supervisors and managers who had been interviewed in the second
phase. The agency’s advocacy officer was also incorporated in this discussion. The
purpose of the focus group discussion was to obtain perceptions of participants
regarding the findings of the study and to explore options for improving the agency’s
M&E system.
The following subsections will now explore these phases in detail.
3.5 Phase 1: Documentary Study
Phase 1 was conducted with the view to address the first two objectives of the study, namely,
to conduct a situation analysis of the M&E system and to assess what works and what does
not work in the implementation of the agency’s design and utilization of the agency’s data.
The student conducted a documentary study of the agency’s M&E documents highlighted in
Annexure A (Data Components). For this phase, a documentary study guide and a data
analysis plan were developed and utilised. This phase was split into two sub-phases as
follows:
Sub-Phase 1a: Locating and Selecting Agency Documents. This sub-phase involved
locating the agency documents that the agency has produced. This was done to evaluate the
comprehensiveness and usefulness of the agency’s M&E system and to identify the gaps in
the current M&E system based on the analysis of the ‘high’ level documents, such as the
59
annual business plan, M&E guidelines and the six indicators highlighted in the first chapter
of this study. Selection was based on availability, as some of the documents were not
accessible. While the student thought these documents would provide valuable insight into
the agency’s M&E system, they were either not in existence or not accessible due to agency
policies on access to confidential information. For instance, the agency’s board documents
related to M&E could not be made accessible to a non-board member.
Sub-Phase 1b: Situation Analysis. This sub-phase involved the analysis of the actual agency
M&E documents indicated in Annexure A. Analysis in this phase helped describe and
illustrate the ‘as is’ situation of the agency’s M&E. Furthermore, analysis in this phase
proceeded by suggesting the ‘what could be’ M&E situation. The ‘what could be’ situation
refers to the M&E information that the agency could generate and produce, and the M&E
documents that the agency could produce.
3.5.1 Population and Sampling
Greeff (2011) describes sampling as a selection process whereby a segment of the population
is used to represent the entire population allowing for inferences about the population to be
made. The population for this first phase of the study was defined as all agency documents
that bear any relationship to M&E. Using a purposive sampling method, the student sampled
for a manageable number of agency M&E documents based on the criteria that the documents
must be relevant to the research questions (refer to Annexure B). In order to reach the sample,
the student sought permission to conduct the study from the Director of the agency where the
research took place (refer to Annexure P - Approval Letter). Once the permission was
granted, the student approached the Foster Care Unit manager who became instrumental in
supplying agency documents relevant to the M&E system. The agency’s advocacy
coordinator was also identified as a resourceful participant who furnished the student with
vital information and other documents. The student sought for documents which were
relevant in informing him of the agency’s current M&E situation. These documents were
gathered in relation to the questions raised in the agency M&E analysis (refer to Annexure B
– Agency M&E Analysis). The student also used his knowledge and competence in M&E to
determine the documents that could capture information about the M&E system of the agency
and began soliciting for them from the said sources.
60
3.5.2 Data Collection Method
Having found the right documents to study, the student started asking questions relevant to
the agency’s M&E situation as captured in the Annexure B. The questions that the student
asked of the documents helped him ascertain what the agency’s M&E is composed of, the
M&E processes at different levels, the documents that are produced at different levels and for
what purpose, and the attitudes of the implementers towards M&E. The student illustrated the
data that he collected using diagrams and sequences (such as those depicted by Annexure H
and Annexure K), which helped him organise and focus his ideas.
3.5.3 Data Collection and Analysis
An inseparable relationship between data collection and data analysis (Schurink et al., 2011,
p. 402) existed during the course of this study. Data collection, according to Rubin and
Babbie (2008), entails the gathering of information required to address a research problem.
For this phase of the study, the tool for data collection was a documentary study. Creswell’s
model of data analysis stipulates that “data analysis is always an on-going process that
routinely starts prior to the first interview” (Creswell, 2009, p. 184). In light of this, the
student collected and analysed data on an on-going basis, hence the study reflected the “spiral
image” or “a data analysis spiral” as suggested by Schurink et al. (2011, p. 403).
In order to manage the data, the student created a case study database comprising agency
M&E documents, divided according to types of documents (e.g. templates, reports, plans)
and clustered according to the level at which they are produced – starting with ‘high’ level
documents (i.e. annual business plan) to ‘low’ level documents (e.g. process notes, plans),
and the documents produced in-between these levels. These documents were filed and
labelled appropriately.
Using a data collection and analysis tool developed by the student (refer to Annexure B),
which specified the dimensions to be studied and a string of questions to ask in relation to
each dimension, it became possible to collect relevant agency M&E data to analyse the ‘as is’
and the ‘what could be’ M&E situation. The student meticulously studied the documents,
read them out in their entirety and accurately made note of key notes arising from the
questions which were asked of the documents.
61
Schurink et al. (2011) describe data analysis as a process of pattern formation involving the
breaking down of collected data and searching for codes and categories. Consistent with this,
the student searched for data categories that were in harmony with each other (e.g. M&E
training, budget allocation, M&E planning) and related to a specific level in the agency M&E
system. The student then coded the data using the open code system. He further interpreted
the data accordingly and, in the process, sought for other plausible explanations for these data
and linkages among them. Appropriate visuals were then developed to present the data as
shown in Chapter 4.
3.6 Phase 2: Semi-structured Interviews
This phase comprised a series of semi-structured interviews purposed to fulfil the second and
third objectives. These interviews helped clarify what works and what does not work in the
implementation of the agency’s M&E design and the utilization of the agency’s data.
Furthermore, they provided useful information concerning the usefulness and appropriateness
of the existing M&E data for organisational performance in rendering community-based
services for OVCs.
Sub-Phase 2a: Interviews with Social Workers. The student individually interviewed four
social workers engaged in delivering specific community-based services for OVCs in Soweto
through the Foster Care Unit.
Sub-Phase 2b: Interviews with Supervisors. Interviews with supervisors were held during
this phase. Two supervisors were separately interviewed.
Sub-Phase 2c: Interviews with Managers. During this phase, interviews with managers were
conducted. Two managers were interviewed.
3.6.1 Population and Sampling
Three populations were sampled for Phase 2 of the study: social workers, supervisors and
managers. The population of social workers was defined as all the social workers working in
62
the Foster Care Unit of the Soweto branch of the selected child welfare agency. The
population comprised 20 social workers. The population of supervisors was defined as all
personnel (from different units) who render direct supervision to social workers and are
involved in the collection and collation of statistical information generated by the agency for
internal and external purposes. The population consisted of eight supervisors. The population
of managers was defined as all unit managers who oversee M&E functions and are involved
in the compilation of annual business plans, progress reports and collation of statistics for
internal and external purposes. The population consisted of four managers.
In this study, which was qualitative, it was not feasible to incorporate the entire population in
the study. Utilising purposive sampling, a non-probability sampling technique, the student
selected a sample of four social workers from the population of social workers. By using
purposive sampling, “a sample of information rich participants” was selected (Struwig &
Stead, 2001, p. 122) and the student was allowed a chance to purposely gather typical and
divergent data (Rubin & Babbie, 2011, p. 355). The social workers who participated in the
study were drawn using the following selection criteria:
- being in the employ of the agency for at least one year;
- not serving a notice of resignation during the month in which the sampling took place;
- delivering services for OVCs in Soweto;
- representative of the different types of community-based services delivered by the
agency;
- willingness to participate in the study; and
- availability to participate in the study.
Besides the social workers, a sample of two supervisors was also selected from the eight
forming the population of supervisors. Since supervisors are key in ensuring the quality of
community-based services delivered for OVCs and are significant in the collection and
collation of statistical information in the agency, the student needed to obtain information
from supervisors who had substantial experience and a good understanding of the M&E
processes in the agency. The following criteria for selection of supervisors were followed:
- being in a supervisor position in the agency for more than a year;
- not serving a notice of resignation during the month in which the selection took place;
- willingness and availability to participate in the study;
63
- experience in supervising social workers who render community-based services for
OVCs in Soweto;
- experience in collecting and collating statistical information in the agency for internal
and external purposes;
- knowledge of the current organisational M&E system and processes; and
- having received training in M&E.
In addition to the supervisors, the student also took a sample of the managers, since it was not
feasible to interview all the managers in the agency (Rubin & Babbie, 2011). The student
needed to tap information from the managers who had a wealth of experience particularly in
the agency and child welfare field in general. The student wanted to understand the M&E
processes, deduce attitudes and fill in gaps concerning the agency M&E system from the
personnel responsible for compilation of some of the M&E documents. The student therefore
purposively sampled two managers based on the following criteria:
- being in a managerial position in the agency for more than a year;
- not serving a notice of resignation during the month in which the selection took place;
- willingness and availability to participate in the study;
- experience in collecting and collating statistical information in the agency for internal
and external purposes;
- managing programmes whose thrust was to provide community-based services for
OVCs in Soweto;
- knowledge of the current organisational M&E system; and
- having received training in M&E.
The two managers who were sampled were the Foster Care Unit manager and the Child and
Family Unit manager.
3.6.2 Data Collection
The student used the predominant mode of collecting data in qualitative studies, which is
one-on-one interviews (Neuman, 2000, p. 34). Broad and multiple questions were utilised to
investigate what works and what does not work in the implementation of the agency’s design
and the utilization of the agency’s data. The student also managed to tap information which
assessed to what extent the existing M&E data was useful and suitable in informing
64
organisational performance in rendering community-based services for OVCs. Semi-
structured interview schedules for social workers, supervisors and managers (refer to
Annexure D, Annexure E and Annexure F) were designed and used by the student to collect
data. These data collection instruments helped the student to verify the findings obtained in
the first phase, fill in the gaps that were not addressed and obtain the lived experiences of the
M&E system from the participants. The interviews were audio recorded and transcribed for
analysis.
3.6.3 Pilot Study
The student conducted a pilot study to determine whether relevant data could be obtained
from the participants using the data collection method and tools devised (Strydom & Delport,
2011, 2011, p. 395). The student acknowledged that the main inquiry could not start unless
confidence was placed on the chosen data collection procedures and tools. Establishing
confidence in the data collection procedures would confirm their suitability and effectiveness
in collecting relevant information, and also ensure trustworthiness. The pilot study was also a
precautionary measure that was taken to avoid any problems and errors from occurring
during the study (Strydom & Delport, 2011, p. 395). The semi-structured interview schedules
were pilot tested with two participants who did not participate in the main research study.
These two participants were recruited using the same sampling method as used in the main
study and they also gave informed consent. The respondents fairly comprehended the
questions and provided relevant responses that covered the scope of the study. The student,
however, made a few amendments to the interview schedules, making them more focused to
elicit relevant information from the participants.
3.6.4 Data Analysis
Data was analysed by the student during and after the interviewing process, as the
relationship between data collection and data analysis is indivisible (Schurink et al., 2011).
Data was analysed in the manner suggested by Creswell (2009, pp. 126-129), according to
the following stages:
Data management. A smartphone was used to audio record the interviews. The recordings
were transferred from the smartphone to a computer. Once transferred to the computer, files
65
were created and labelled appropriately. These files were subsequently transferred to compact
disks (CDs) which were also labelled appropriately. The student listened to the interviews
carefully and transcribed them into written text.
Reading and writing memos. Having organized and converted the data into a suitable
format, the student read the entire transcripts multiple times. Key issues relevant to the goal
and objectives of the study were noted. At this phase, minor editing was performed on the
transcripts, such as correcting spelling errors and inserting missing words. The student wrote
notes, called memos, on the transcripts, to capture his preliminary thoughts, impressions and
hunches, during this early reading of the data.
Describing, drawing patterns, themes and coding. During this stage of data analysis, the
student identified various categories of information that were prominent in the interviews.
This gave the student a chance to classify information into themes and sub-themes to make
the data manageable. Using an open coding system (Creswell, 2009), the student identified
and labelled ideas, concerns, attitudes, and processes and procedures as related to the
dimensions which were being examined. Data was interpreted accordingly and explanations
and linkages between and among data were formed. The student made use of Atlas.ti – a
qualitative computer-based software – to upload the interview transcripts. Since themes and
subthemes were already determined, the student created them in Atlas.ti and created linkages
with quotations that substantiated the themes. Additional memos and notes were also created
as relevant to the themes during the data analysis phase.
Representing and visualising. The student developed a diagrammatic visual, which was
used to present the data depicting various themes. This will be discussed and substantiated
with verbatim citations in the following chapters of this study.
3.7 Phase 3: Focus Group Discussion
In the third and final phase of data collection, a focus group discussion was conducted to
obtain perceptions of participants regarding the findings of the study. The main purpose was
to verify the findings and explore options, which were triangulated with the literature. Only
one carefully planned and systematically facilitated focus group discussion was sufficient for
66
the intended purpose. It was conducted in a permissive and non-threatening environment and
it lasted for approximately 90 minutes, which is considered to be appropriate for such a
discussion (Strydom, 2011b, p. 361). The findings of the study obtained up to the end of
Phase 2 were presented and options for improving the M&E system of the agency were
explored. The focus group discussion also served to generate recommendations for an
appropriate and effective M&E system for community-based services delivered to OVCs in a
child welfare setting.
3.7.1 Population and Sampling
The participants in this focus group discussion were the two managers, two supervisors and
four social workers, all of whom had been interviewed by the student in Phase 2. In addition,
the advocacy coordinator, who uses statistics for lobbying and influencing policy and who
contributed to the selection of appropriate documents for Phase 1, was included. In total, the
focus group discussion comprised nine participants, which was a good number to generate a
discussion in an environment where participants were not feeling overcrowded (Morgan &
Krueger, as cited in Strydom, 2011b, p. 366). Opinions of a diverse group of people who are
actively involved in service delivery were obtained (Strydom, 2011b, p. 367).
3.7.2 Data Collection Method
The focus group discussion was utilised in eliciting data from the participants in this phase.
This data collection method allowed for several participants to be questioned systematically
and simultaneously (Rubin & Babbie, 2011, p. 467), verifying the findings and exploring
options of what participants thought would constitute an appropriate and effective M&E
system for community-based services for OVCs delivered in a child welfare setting. The
focus group discussion also created “a fuller, deeper understanding of the phenomenon being
studied, and stimulated spontaneous exchanges of ideas, thoughts and attitudes in the security
of being in the crowd” (Greeff, 2011, p. 374). As highlighted earlier, there was an inseparable
correlation between data collection and data analysis. Data was recorded using a smartphone,
thus the focus group discussion was captured word for word whilst communicating, listening
and probing the participants (Rubin & Babbie, 2011). The audio recording was subsequently
transcribed. Sketchy notes, which were unobtrusively written in form of words and phrases,
were also made by the student during the focus group (Creswell, 2009). This did not disrupt
67
the session and it enabled the student to keep abreast with what was happening in the focus
group (Rubin & Babbie, 2011).
3.7.3 Data Collection Tool
In order to effectively facilitate the focus group discussion and extract in-depth information
from the nine participants, the student made use of a questioning route (refer to Annexure G).
The student presented the findings of the study to date starting with the current M&E
situation and gaps and then made a presentation of what works and what does not work for
the agency concerning the M&E system. Findings relating to the current situation and gaps
were presented sequentially under the following headings – overall M&E policy and
guidelines; organisational structure; knowledge of the principles governing community-based
services for OVCs; community-based services; M&E tools and processes within the agency;
M&E data processing, analysis and use; reporting; and training. In between, the participants
were allowed to spontaneously respond to the findings and make necessary comments and
contributions. Thus, only a few open-ended questions were formulated, allowing the focus
group participants to give detailed responses of the findings that were presented to them.
Given the methodical manner in which the focus group was conducted, the questioning route
served only as a guide, with follow-up questions being asked during the course of the
discussion. Participants naturally responded to the findings of the study and commented
appropriately, thereby generating a discussion in which some of the questions on the
questioning route were answered before they could even be asked.
3.7.4 Data Analysis
Since the focus group discussion was well planned, focused and highly structured, it speedily
generated data from the participants (Rubin & Babbie, 2011). Thus, data analysis was
relatively easy and straightforward.
Data analysis involved thoroughly reading the transcripts and writing memos and comments
(Creswell, 2009), using a qualitative data analysis software entitled Atlas.ti used in the
previous data analysis phase (semi-structured interviews). Since presentation of findings was
systematic and in accordance with specific themes identified by the student and described in
the following chapters, analysis concentrated on the context and extensiveness of comments
68
made (Greeff, 2011, p. 373). In essence, data analysis focused on whether comments
validated the findings, disputed them or made further contributions and generated
recommendations. Themes were predetermined and identified prior to data analysis, which
made it easy for the comments to be classified according to specific themes.
3.8 Strategies for Ensuring Trustworthiness
In order to enhance trustworthiness of this study as a whole, strict measures were followed.
Burns and Grove (2009, p. 53) highlight that scientific rigour in research entails “openness,
relevance, epistemological and methodological congruence, scrupulous adherence to a
philosophical perspective, thoroughness in collecting data and consideration of all data in the
analysis process and the researcher’s self understanding”. Given this explanation, the
following are measures that the student followed to ensure trustworthiness of the study.
3.8.1 Credibility
Correct operational measures and specific procedures required in qualitative studies were
employed to enhance the credibility of the data (Miles & Huberman, 1994, p. 34). Thus, the
student was meticulous in the correct application of data collection and data analysis
procedures widely used in qualitative studies, thereby ensuring the credibility of the findings
(Miles & Huberman, 1994). The student also thoroughly read the M&E documents collected
in Phase 1 data collection and the interview transcripts (Creswell, 2009, p. 192), which
enabled him to capture the accurate descriptions as captured by the agency and reported by
the participants. Furthermore, the population was accurately identified and described
(Schurink et al., 2011, p. 386).
3.8.2 Confirmability
According to Miles and Huberman (1994, p. 34), “detailed methodological descriptions
enable readers to determine how far the data and constructs emerging from it may be
accepted.” In light of this, the student ensured confirmability by providing a detailed
description of the methodological process (see Section 3.3 above) to substantiate that the
research findings emanated from the experiences and knowledge of the participants, rather
than the student’s assumptions (Schurink et al., 2011, p. 421). The role of triangulation,
69
employed in this study, in promoting confirmability can, at this stage, be emphasized, as it
served to reduce the bias of the student. Also key to confirmability is the student’s admittance
of his predispositions through the reflexivity statement given in Section 3.9 below.
3.8.3 Transferability
Transferability is concerned with the extent to which the findings of the study can be applied
to other situations (Shenton, 2004). This study was conducted within the confines of the
developmental social welfare theoretical framework. Data was collected using semi-
structured interview schedules and through documentary study, and was analysed using the
guidelines of Creswell’s (2009) model. In addition, this study provided “sufficient contextual
information about the research site” (Shenton, 2004, p. 69) to enable transferability. As such,
a replication of this study, conducted within the same parameters, should produce similar
findings as produced by this study (Schurink et al., 2011).
3.8.4 Member Checking
Member checking was a critical technique that was employed in this study to enhance
trustworthiness. In fact, Guba and Lincoln (as cited in Shenton, 2004, p. 68) contend that
“member checks are the single most important provisions that can be made to bolster a
study’s credibility.” The student interpreted the information that was obtained from the
documentary analysis and the semi-structured interviews and then checked with the
participants during the focus group discussion to see if the interpretation and thematic
analysis was consistent, correct and congruent with their experiences (Greeff, 2011, p. 372).
3.8.5 Dependability
Shenton (2004) stresses that there are close ties between credibility and dependability,
arguing that, in practice, a demonstration of the former may serve to ensure the latter. This is
achieved through the use of “overlapping methods” (Shenton, 2004, p. 71) such as those used
in this study – the document analysis, individual interviews and focus group. The student
made use of peer scrutiny (Shenton, 2004, p. 11) to enhance the dependability of the study.
Colleagues (qualified social workers with master’s degrees) who have no connection to the
study were asked to examine and assess the accuracy of the methodology, the findings,
70
interpretations and conclusions. The feedback that was obtained gave the student a fresh
perspective and enabled the student to refine the research with real objectivity. Moreover, the
student worked under the guidance of knowledgeable and capable supervisors, which helped
increase the dependability of this study, as the student could refine his methods, develop
greater explanations of the research design and strengthen his arguments. In order to ensure
dependability more directly, the processes within the study, that is the research design and
implementation, data collection methods and a reflective appraisal of the study, were reported
in great detail (Shenton, 2004, p. 71).
3.8.6 Triangulation
The study involved documentary analysis, semi-structured interviews and a focus group
discussion, which form major data collection strategies for much qualitative research
(Shenton, 2004). The use of these different methods served to offset their individual
limitations whilst exploiting their respective benefits (Burns & Grove, 2009). Also, the use of
these different data collection techniques enabled the study to generate findings from multiple
perspectives, which enhanced the trustworthiness of the study.
3.9 Reflexivity Statement
My curiosity for monitoring and evaluation has evolved over the course of several years.
Having previously worked in research institutes, in the field of human capital development,
and currently working in the child welfare sector, I have become increasingly aware and
more inquisitive about the ‘so what’ question (Rabie, 2011) that M&E tends to ask
particularly in social service delivery.
Since I have worked for a considerable period in the child welfare sector, I have realized that
M&E is hardly used as a tool for planning, guiding resource allocation, improving the quality
of service delivery and demonstrating results (World Bank, 2002). Rather, emphasis is
usually placed on M&E when there is need to attract funding or in situations where there is a
demand for accountability by funders or stakeholders. More often than not, weight is placed
on processes and outputs. The design of programmes and scope of service delivery, in the
NGO sector, are thus determined by societal trends and service areas where there is potential
71
for attracting better funding. Demands for accountability and regular provision of M&E
information to government are known to be deterrent factors driving professionals from child
welfare organisations (Lawrence et al., 2013).
High staff turnover, bureaucratic pressures, shortage of resources and high caseloads are
characteristic of child welfare organisations (Lawrence et al., 2013; Schmid, 2013). Amid
these unyielding concerns and excessive work demands placed on social workers and
managers, child welfare agencies are left with no choice but to prioritize (Lawrence et al.,
2013). Undesirably, M&E does not get its due importance in the prioritization process.
Resultantly, focus is placed on service delivery, which is accounted for in terms of
quantitative output indicators. M&E training and functions are thus relegated to the level of
aspects that are time-consuming and processes that deter service delivery.
I, however, approach this study as a qualified social worker and a social work supervisor who
has acquired training in M&E. I have gained experience in rendering direct social work
services for OVCs, but now I supervise practitioners who render such services in the agency
where the research is being conducted. While I am appreciative of the role that M&E plays in
improving the quality of service delivery and demonstrating results at outcome and impact
level, I concede that social work education per se, does not adequately equip social workers
to effectively engage in M&E. I acknowledge the knowledge gap and incompetence of social
work practitioners to engage in evaluation activities, which I attribute to low worker
compliance with evaluation (Lawrence et al., 2013, p. 6).
I am conscious of the fact that child welfare agencies have some form of M&E systems.
However, I come to this research having observed that these systems are, to some extent,
inappropriate, ineffective and unappreciative of the actual role and value of M&E. The
question in child welfare is probably what the most useful M&E system is and how such an
M&E system could be effectively implemented without exerting unwarranted pressure on
practitioners. Patton (2004) suggests an M&E system that is utilization focused. The World
Bank (2013) suggests an M&E system which is participatory and focused on creating buy-in,
ownership and enhancing sustainability. Rabie and Cloete (2011) accentuate that M&E
should be based on scope, philosophy and design. While these ideas for M&E have been
critiqued by many, for this study, I maintain that M&E systems in the child welfare sector
should be theory-based, participatory and utilization focused. This is significant in improving
72
policies that govern the sector, enhancing the commitment of practitioners to evaluation and
ensuring that M&E results are fully utilised.
Therefore, as I interviewed social work practitioners, supervisors and managers regarding
their M&E experience in the agency, I bring with me perceptions of how M&E could be and
experiences (through training) that make me an ardent advocate for specific M&E principles
and practices. My educational background and my experiences (both within and outside of
the agency under study) serve to guide this research into the most appropriate M&E system
for organisations rendering community-based services to OVCs.
3.10 Ethical Considerations
The following is a discussion of the ethics that were taken into account during this study.
First, is a discussion on how the agency was identified and selected to be the case study.
Second, the student delves into detailed explanations of specific ethical issues that were
respected during this study.
The student is in the employ of the agency under study. As such, the agency was identified as
the research site. Permission was requested for the study to be conducted in this agency
through a formal written request submitted to the management of the organisation. The
request for permission letter (refer to Annexure O) explained the title, goal and objectives of
the study. Furthermore, it explained what the study entailed in terms of the duration,
participants, dissemination of research results and incentives. The intention to conduct this
study was discussed between the manager of the Foster Care Unit and the director of the
agency. Subsequently, agreements were reached (together with the student) related to
anonymity, privacy, confidentiality of agency information and avoidance of harm to
participants. An approval to go ahead with the study was then issued (refer to Annexure P).
The following sections address, in detail, specific ethics as related to the individual
participants and the agency as a whole.
73
3.10.1 Avoidance of Harm
The study aimed to evaluate the appropriateness and effectiveness of the M&E system in a
child welfare agency rendering community based services for OVCs and to make
recommendations on how the system might be improved. The probable effect was that some
emotional issues could be triggered due to the demands placed on social workers to engage in
M&E duties amidst resource constraints and the pressure to deliver comprehensive
community-based services for OVCs. As such, the student adhered to the principle of
avoidance of harm, so that the study did not leave participants in psychological distress
(Taylor, 2000, p. 8). The student thoroughly informed participants of the potential impact of
the study and gave each participant an opportunity to withdraw from the study if they so
wished in order to curtail the likelihood of harm resulting from this study (Rubin & Babbie,
2011, p. 77). Care was taken not to put the name of the agency into disrepute during the
evaluation of the agency’s M&E system.
3.10.2 Informed Consent
As highlighted above, the student first requested for permission from the management of the
agency to conduct the study explaining the purpose and what the study entailed. The student
only proceeded once permission was granted. The student also provided adequate information
to the research participants and clarified the aim and objectives of the study. The student
ensured that participants comprehended the purpose of the study and that they made a
reasoned and free decision about their participation in the study (Taylor, 2000, p. 7).
Furthermore, the student provided information about procedures to be followed during the
semi-structured interviews and the focus group discussion outlining the possible advantages
and disadvantages of the study. The credentials of the student were also described to the
participants (Strydom, 2011a, p. 117). No participant was coerced into taking part in the
study. Rather, the student adhered to the principle of voluntary participation (Babbie, 2001, p.
470) and asked each participant to sign an informed consent form (refer to Annexure C) prior
to the first interview. The student repeated key information concerning the informed consent
form and clarified any uncertainties to the participants.
74
3.10.3 Avoidance of Deception
No form of deliberate deception or intentional withholding of information was made by the
student (Strydom, 2011a, p. 118). The management of the agency were given a clear and
exhaustive explanation of what the study entailed before they decided to give approval of the
study. There was no hidden agenda in this study and every participant was given adequate
and accurate information necessary for them to decide their participation (Taylor, 2000, p. 9).
3.10.4 Privacy, Confidentiality and Anonymity
The student respected the agency’s right to confidentiality and was sensitive in keeping
documented material and information about the agency private. Documents that were
gathered during the documentary analysis phase as well as information that was obtained
during semi-structured interviews with individual participants was thus handled with respect
to privacy and confidentiality.
Since the participants knew each other and saw and heard each other during the focus group
discussion, the student was not able to assure anonymity. However, the student asked the
participants to treat the information shared during the focus group discussion with the
strictest confidence, as was explained in the informed consent form. Participants were made
aware that a smartphone and field notes were going to be used to record data and their
consent to use such data collection tools was obtained. Participants were also brought to the
awareness that a research report was going to be compiled by the student and submitted to the
University of Johannesburg for academic purposes. It was also highlighted that a possible
scientific article would follow, but assurance was given that no information that would
identify them (specifically, personal identification details) would be included in the
transcriptions, research report and/or any other publications to follow. It was also brought to
light that raw data (including transcriptions and informed consent forms) would be securely
stored in accordance with University of Johannesburg requirements.
75
3.10.5 Actions and Competence of the Student
Strydom (2011a) highlights that poor interviewing skills and incompetence of the student
may result in the collection of data that may have little use. The student was competent
enough to undertake the study, since he has adequate interviewing skills developed through
his previous employment and participation in social research as a research assistant.
Additionally, he has successfully completed a module on research methodology in his
undergraduate studies and has attended relevant post-graduate workshops that equipped him
with the necessary skills to conduct this study. Moreover, the student possesses a fairly
developed understanding of the phenomenon he wished to study, which aided in the research
process (Strydom, 2011a, p. 115). He is well versed in group facilitation techniques and skills
and possesses the necessary communication skills (Greeff, 2011, p. 368). Finally, he
conducted this study under the supervision of two experienced supervisors.
3.10.6 Collaboration with Contributors
Proper credit was granted to the people who contributed to this study, that is, the participants,
the management of agency where the research took place and the supervisors allocated to the
student by the University of Johannesburg (Strydom, 2011a, p. 25).
3.10.7 No Compensation
No compensation or incentives were offered to the agency for approving this study. Also, no
individual participants were rewarded for being part of the study. The interviews were
conducted at the agency premises during the week. Therefore, the participants did not need
reimbursement for transport costs. There was no payment requested by the agency for the
student to use their facilities and to involve their staff in the study (Greeff, 2011, p. 371).
3.10.8 Publication of Findings
The research findings were presented in an accurate, objective and ethically sound manner to
participants in the study. Information was conveyed in clear and unambiguous terms
(Strydom, 2011a), acknowledging the work of others to avoid plagiarism (Babbie, 2001). The
limitations and shortcomings of this study were also acknowledged. The research findings
76
will possibly be submitted for publication in a scientific journal and as a form of recognition
and expression of gratitude, the student will objectively inform the participants and the
management of agency where the research took place about the findings (Strydom, 2011a, p.
126).
3.11 Conclusion
The purpose of this chapter was to illuminate on the research design and methodology used in
this research, which includes the research site, sampling, data collection instruments and
procedures, data analysis, measures to ensure trustworthiness and ethical considerations.
Particular attention was given to the three phases of the study, the findings of each of which
will be presented in separate chapters. The following chapter will present a discussion of the
Phase 1 research findings, that is, findings emanating from the documentary analysis.
77
Chapter Four: Phase One Findings: Documentary
Analysis
4.1 Introduction
The preceding chapter illuminated the procedures that were followed in fulfilling the
objectives of the study. The research methodology was described by illustrating how the
research information was obtained and what tools, designs and samples were used to collect
data. In this chapter, the findings of the respective phases of data analysis are presented and
discussed.
The first phase in this chapter presents and discusses findings that relate to the ‘as is’
situation of the M&E system of the agency under study. These findings emanate from the
analysis that was conducted by the student of the relevant M&E documents produced by the
agency itself. The student presents these findings in illustrative and descriptive ways
portraying how the agency’s M&E system is constructed.
The second phase in this chapter presents and discusses the findings emanating from the
semi-structured interviews, first, conducted with the social workers and, second, conducted
with managers and supervisors in the agency.
The third and last phase explores findings derived from the focus group discussion conducted
with the aforesaid social workers, supervisors and managers. Findings in this chapter are
presented in the form of themes and sub-themes substantiated by verbatim statements, which
reflect the situation of the agency and support the views of the participants and those of the
student. Literature is used to locate this study’s findings within the broader body of national
and international research and theory.
Findings in this first phase present a situation analysis of the agency’s M&E system in
fulfilment of the first objective of this study. The student therefore commences this section by
giving an overview of the agency’s programme structure. This is vital in understanding the
agency’s M&E system as it exists in practice, that is, how it is constructed and which
78
components it is comprised of. The student also strived to gain an understanding as to why
the agency’s M&E situation is the way it is. Data gathered in this phase was subsequently
triangulated (in Phases 2 and 3) with information gained from the semi-structured interviews
and the focus group discussion with the social workers, supervisors and managers involved in
rendering services and monitoring and evaluating community-based services for OVCs in
Soweto.
4.2 Overview of the Agency’s Programme Structure
It is significant to note at the onset that the agency does not have a separate M&E unit. As
such, the M&E function is embedded in the programmes of the agency where, at each level in
the organizational structure, employees are responsible for various M&E activities. For
instance, the collection of M&E data, reporting of M&E issues and planning for M&E are
activities carried out by different employees who are at different levels within the hierarchical
structure of the agency. At each specific level, there are activities conducted and documents
produced that relate to the basic M&E components, which are inputs, processes, outputs,
outcomes and impact (Rabie & Cloete, 2011). To clearly illustrate this description, the
student diagrammatically presented his findings (refer to Annexure H).
The way in which M&E is mainstreamed into the functioning of the agency is characteristic
of bureaucratic organizations with “the functional team” structure (Rabie, 2011, p. 55). Such
an M&E system relies on co-ordination of activities by bureaucratic means through detailed
task specification and occasional ad hoc meetings (Rabie, 2011, p. 55). Mainstreaming M&E
into the functioning of the agency makes M&E a core business and provides minimum
disruption to the traditional bureaucratic structures and overall operations of the organization
(Rabie, 2011). However, the shortcoming of doing so lies in that there is no specific expertise
in the agency and no one giving focused and dedicated attention to M&E. Consequently,
managers may neglect M&E concentrating on service delivery.
The overall M&E system of the agency is informed by guidelines or policies stipulated by the
Department of Social Development (DSD) and those set down by the agency itself.
Moreover, the six indicators that the agency is tracking (namely, distribution of services,
quality of services, quantity of services, time frame indicators, accessibility, and issue and
79
response specific indicators) are pivotal in informing the overall M&E system. Since the
DSD provides the agency with the greater part of the funding required in running OVC
programmes, the agency is obliged to align itself with specific objectives contained in the
business plan. An integration of the DSD and agency guidelines for M&E is what gives the
agency the mandate to render community-based services for OVCs in Soweto. Documents
produced at the ‘highest’ level of the agency structure stipulate the principles which guide
decisions in order to reach an overall outcome (Rabie, 2011). The M&E guidelines are
translated by the agency into programmes and plans in order to think ahead, coordinate
activities, and organize resources towards a common goal (Rabie, 2011).
The agency then puts the programmes and plans into action through various community-
based services for OVCs. These services are rendered through various specialized units
within the agency. The units serve as entry points where intake, referrals and transfer of cases
are conducted (refer to Annexure H). Given the situation that M&E is embedded within the
agency structures and that the agency has specialized units to fulfil its purposes, there is
therefore ‘institutional fit’ between the agency’s structure and purpose (Ackron, 2013).
Each unit is headed by a manager who is responsible for overseeing the M&E function and
reporting on various objectives and indicators within that unit. The manager has a pool of
personnel including social workers responsible for rendering direct services to clients.
Managers also conduct various M&E activities and collect data that is useful for M&E
purposes.
Under each unit manager, there is a team of supervisors who enable, ensure and enhance
humanistic, competent and independent social work practice (Kadushin & Harkness, 2002, p.
23). Their educational and supportive functions promote ethical and effective job
performance within the agency (Kadushin & Harkness, 2002). However, it is the hierarchical
administrative role of supervisors which is key within the M&E system of the agency. This
role entails management of resources, setting objectives and priorities, clarification of roles,
planning and assignment of work, review and evaluation of supervisees’ work, being
accountable and responsible for supervisees’ work (quality and quantity of services
rendered), as well as collating various data elements.
80
Under the supervisors are the teams of social work practitioners who render direct services to
beneficiaries. Their role in the M&E system is to implement the programme (through service
delivery) and collect various M&E data emanating from the services delivered. In collecting
M&E data, social workers use various tools and templates that have either been designed by
the agency or have been prescribed by the DSD.
The agency has a bureaucratic and hierarchical structure which allows for the collection of
various M&E data using a bottom-up approach with quality checks being employed at
different levels. An analysis of the programme structure reflects inputs, processes and outputs
as major components measured by the agency. The following section will meticulously
explore the findings obtained from the specific documents analysed by the student.
4.3 Results Emanating from the Documentary Study
This section presents the findings that were drawn from the study of the documents and
statistical information that are produced by the agency itself. Using the sampling method
described in Chapter 3, the student analysed the documents listed in Annexure A. In order to
make conceptual and logical sense in the analysis of the documents and presentation of the
findings, the student created a cluster and sequence mapping of the agency’s M&E system
(refer to Annexure K). As illustrated, the agency’s overall M&E system is governed by
procedures, principles and objectives articulated in the ‘high’ level documents (that is, the
annual business plan and M&E guidelines from the DSD and the agency itself). Coupled with
these documents, are a set of indicators, which inform goal achievement and the agency’s
performance relative to set objectives (UNAIDS, 2010, p. 77). This forms the pinnacle of the
agency’s M&E system, which is useful in informing M&E processes at the point of service
delivery (which is the ‘lowest’ level of the agency’s M&E system). Processes which ensue at
the supervisor and line manager level, relate to the translation of data gathered by service
delivery staff into M&E indicators. Thereafter, the agency generates M&E reports, which
give account to stakeholders about the M&E indicators and, by and large, inform governance.
The following table is based on the aforementioned clustering and sequencing of the agency
M&E system and it illustrates the rationale for clustering specific documents which the
student studied. The terms ‘high’ and ‘low’ used by the student in the cluster and sequence of
81
the documents does not imply that they are of lesser or greater importance. Rather, they refer
to governance and decision making level (‘high’ level) and the operational level (point of
service delivery) of the overall system.
Table 4.1 Description of Documents Studied
RATIONALE FOR
CLUSTERING
SPECIFIC DOCUMENT/ COMPONENT ANALYSED
‘High’ level documents governing
M&E processes Business plan; DSD and agency M&E guidelines
‘Low’ level documents used to
collect raw M&E data Templates; Daily diary; Process notes; Reports; Plans
Processes and tools used to
translate data into M&E
indicators
Supervisors and managers collating data; Soweto Care System
Management of M&E processes Manager’s minutes
M&E reports of indicators that
inform the programme
Six-monthly progress reports; Annual report; Statistics for internal and
external stakeholders (including DSD).
4.3.1 ‘High’ Level Documents Governing M&E Processes
It is important to highlight that the student endeavoured to find documents in the agency that
stipulate the M&E guidelines as prescribed by the DSD. Also important for the student to
locate, were the agency’s own guidelines for M&E. The student, however, failed to locate
them as they were not available in the organization. Furthermore, there were no agency
documents that were available to the student which speak to the six indicators that the agency
is said to be tracking. It appears that these documents do not exist.
This first finding is concerning, as it suggests that the agency’s M&E system is operating
without coherent and strategic direction and guidelines. The student therefore has questions
about how M&E indicators and processes are determined and prioritised, how agency staff
from managers down to practitioners know what is expected in terms of M&E, how M&E
data are utilised to improve programme performance, and how M&E data are utilised to
review and adjust agency strategy.
82
The only high level document that could be obtained and analysed was the annual business
plan. Principally, the annual business plan provides detailed information related to the major
objectives of the programme to achieve throughout the year, and the strategies to be
employed in order to achieve each of the objectives. It stipulates the expected outcomes and
indicators to monitor, which are fundamental elements of any M&E system (Rabie & Cloete,
2011). It also contains information about the team attempting to reach those objectives, the
target group, programmes and services, and how to execute the services in terms of staffing,
budgets and infrastructure.
The annual business plan is in the form of an electronic template which is compiled by the
unit manager by the beginning of each financial year (i.e. by 1 April). Each manager submits
their annual business plan to the respective deputy director in charge of the overall unit (that
is to say there is a deputy director to whom each unit manager reports). These deputy
directors proofread the annual business plans to ensure that the agency is not committed to
doing things beyond its scope or to services that will exert strain on the staff. Once this is
done, the annual business plans are submitted to the DSD for purposes of monitoring how the
agency intends to utilize the funding. The unit managers, accompanied by a finance board
member, are later on summoned by the DSD to attend a panel meeting in which they will
justify their annual business plans and/or request for more funding. In this panel, the DSD
gives managers feedback regarding the previously submitted annual business plan. Once the
business plans are approved, the agency directors and a board member sign a service level
agreement. This entails the DSD clarifying the services they will fund for the year and
spelling out the reporting requirements, which include submission of monthly ‘statistics’ and
the six-monthly reports. The agency will underscore their commitment to the delivery of
services and reaching objectives as highlighted in the business plans.
Since this study focused on the Foster Care Unit, it analysed the six objectives of the unit,
which are stated in the Business Plan as follows:
1. To recruit, screen and train 225 additional prospective foster families in order to be
able to place an estimated 225 children in alternative care.
2. To provide supportive, monitoring and treatment services to 800 children placed in
foster care in order to ensure that placements are in the best interests of the children
83
and to provide the same services to 200 children placed in residential care facilities in
conjunction with these facilities.
3. To render therapeutic group services and to build up self-esteem, confidence and self-
worth in 300 children placed in foster care through using this method.
4. To provide supportive and empowering services to 800 foster families and to enable
them to provide suitable care for the children.
5. To place 45 children, who are either in residential care or in foster care back with
their biological families and to support and monitor these placements.
6. To motivate and encourage 50 foster children to improve on their academic, sporting
or cultural activities, for which they are funded by the Bridgman Foundation for
Educational Excellence and 10 young adults who are funded for their tertiary
education by the Rheinalt Jones Trust.
Each of these objectives has the following components;
Activities: Activities refer to services to be rendered and actions to be taken to
translate inputs into tangible outputs.
Performance indicators: Performance indicators are percentage or numerical values
of certain aspects to be achieved.
Outcomes: According to the business plan, outcomes refer to products or services
delivered as a result of the activities (outputs).
Personnel and resources: These are human, capital, material and physical resources
required to do the work (e.g. supervisors, social workers, tele-communications
equipment, cars, printers, etc).
Location: Location refers to specific places that practitioners need to visit (e.g.
schools, clinics, foster care homes, Child & Youth Care Centres - CYCCs).
Total costs: Total costs refer to the allocated percentage of the total budget and the
actual value in monetary terms.
The following is an example of how the annual business plan captures the objectives and
information on how these objectives are monitored:
Objective 3. To render therapeutic group services and to build up self-esteem, confidence
and self-worth in 300 children placed in foster care through using this method.
84
ACTIVITIES PERFORMANCE
INDICATORS
OUTCOMES PERSONNEL AND
RESOURCES
LOCATION TOTAL
COSTS
1. Group work
whereby
therapy,
enrichment
and
empowerment
programmes
are provided.
- No. of children at
year end showing
improved levels of
independent living
skills as assessed by
social workers
- No. of children at
end of school year
showing goal-
directed progress as
assessed by the social
workers
75 foster
children
undergoing
life skills
programmes
and/or career
guidance
every quarter.
- 1 manager (10% of the
time)
- 3 supervisors (10% of
their time).
- 14 social workers and 1
social auxiliary worker
(15% of their time).
- 5 vehicles
- Administrative
assistance e.g. typing – 2
typists (5% of their time).
- Rental, parking, petrol,
photocopier, computers,
printers etc.
Head office
or any other
available
venues
within the
areas of
operation.
10% of the
total
budget
R756 394
The illustration above describes how the agency is using the annual business plan as a tool to
plan the measurement of objectives in terms of input indicators (‘costs’), processes
(‘activities’), output indicators (‘performance indicators’) and outcome indicators
(‘outcomes’). Information in the annual business plan is being used by the agency to report
on outputs and expected outcomes as set by the agency. However, information in this ‘high’
level document could be used as inputs to inform how services at point of delivery level
(‘lowest’ level of the M&E system) can be improved to achieve expected results. This
document is currently being used as a management and administrative tool reporting mainly
on inputs, processes and outputs. It has potential to be used as a tool for measuring outcomes
and impact. This underutilization of M&E information does not assist the agency in
improving its implementation of the M&E system. The knowledge gained is inadequately
filtered through the specialised units to improve service delivery.
Other than that, the annual business plan also provides guidelines for monitoring and
evaluating programmes. Based on the 2012-2013 business plan analysed by the student for
the abovementioned unit, the following is an example of the approach that informs the
agency’s M&E plan:
“The Foster Care and Reunification Services Department will measure
its performance on each of the objectives, with their targets, by
comparing the activities undertaken against the performance indicators
and the expected outcomes. Taking the challenges into account any
shift in focus will then be made in order to achieve objectives where
necessary.”
85
The M&E plan above underscores the importance of set targets and expected outcomes in
measuring whether the objectives were achieved. This M&E plan is, however, imprecise in
explaining the exact steps that will be employed to monitor the overall programme and
evaluate whether the overall goal is achieved or not. It is not meticulous and comprehensive
enough to answer the ‘who’, ‘what’, ‘when’ and ‘why’ questions of M&E. This appears to
the student as a lack of emphasis on M&E; a situation which leaves gaps in the agency’s
M&E system whereby staff responsible for implementing M&E are not provided with
guidance for continuous data quality improvement.
USAID (2010, p. 10) states that the “rule of thumb is that 5-10% of a project budget should
be allocated for M&E”. It is a noteworthy finding that the total budget received by the agency
from DSD is committed towards service delivery and that there are no funds allocated in the
annual business plan towards the training of staff in M&E per se. In-house training focuses
on enhancing service delivery by improving social workers’ knowledge and skills. The
annual business plan clearly emphasizes the need to “identify skills and knowledge gaps”
among social workers with the view of creating personnel “who will improve service delivery
to clients and children” (Annual Business Plan, 2012/04/01 – 2013/03/31, p. 6). Thus, while
M&E is mainstreamed into the life of the organisation; it has no dedicated M&E personnel
and M&E functions are not financially supported by the business plan.
The student discovered that the annual business plan incorrectly refers to outputs as
outcomes. The following examples of listed ‘outcomes’ as drawn from objective 1 in the
annual business plan give clarity to this assertion:
“Recruitment of 32 foster parents each month”
“Screening 32 prospective foster parents each month”
“Training 19 prospective foster parents per month”
Essentially, these items listed above are reflective of the targeted outputs (not outcomes)
since they do not speak to the difference that services rendered make to the beneficiaries.
This, conceivably, implies the need for the agency to train its staff and management in M&E
theory and terminology.
86
4.3.2 ‘Low’ Level Components: Forms/Templates for Collecting Raw M&E Data
Given that the annual business plan is a ‘high’ level document which has implications on how
M&E processes are conducted at the point of service delivery, the student proceeded by
analysing forms and templates that are used by social workers to collect raw M&E data.
Using the sampling method described in Chapter 3, the student selected reports, plans,
templates for internal statistical data, templates for collecting statistical information
submitted to the DSD and the daily diary.
The agency’s social workers render a range of community-based services to beneficiaries on
a daily basis (refer to Annexure I for a listing of these services). Practitioners capture records
of the services rendered per case (client contacts or process notes) using a computer-based
programme called the Soweto Care System (SCS). Process notes capture the following
information: contact type (e.g. home visit, school visit), date, duration, purpose, and
professionals involved (e.g. teacher, therapist, presiding officer). In the process note, the
practitioner captures a summary of the main activities conducted during the contact including
a record of the issues discussed. The practitioner also records their observations and remarks
about certain aspects they would have found striking during the contact. The practitioner then
makes an evaluation of the services rendered and devises their next plan of action. Process
notes for each client are then printed and filed in the respective client’s file.
Process notes are important to the agency as they provide data for output indicators (quantity
and types of services rendered per individual client). Nonetheless, they are being
underutilised, as they are also a rich source of information useful in tracking the six
indicators, highlighted earlier, that the agency is tracking. If properly captured using the SCS
and analysed, process notes could be utilised to inform the agency about other indicators
being tracked such as the quality of services being rendered, the response of the agency to
specific problems, and time frames between the determination of need and the actual service
delivery. This would, however require greater uniformity in the structure in which process
notes are recorded by social workers and more optimal utilisation of the SCS to generate
these additional indicators.
In addition to completing process notes, social workers also compile early intervention plans
(EIP), individual development plans (IDP) and care plans (CP). These plans are based on
87
templates that have been designed by the agency which the social worker completes
manually. There are no electronic or computer-based versions of these templates. They are
filed in individual client files and are checked by supervisors when the file goes for
supervision. These important documents are not utilised for M&E purposes.
The EIP is a strategic map to foster the development of a child in the Circle of Courage areas
of belonging, mastery, independence and generosity (Brendtro & Larson, 2006). The IDP is a
tool which aids a social worker to develop personal goals, targets and commitments in respect
to the needs of each child. These goals should be measurable and time-bound drawing on the
strengths of the child. The EIP and IDP relate closely to objectives 2 and 4 of the annual
business plan, namely “to provide supportive, monitoring and treatment services to 800
children placed in foster care in order to ensure that placements are in the best interests of the
children and to provide the same services to 200 children placed in residential care facilities
in conjunction with these facilities” and “to provide supportive and empowering services to
800 foster families and to enable them to provide suitable care for the children”.
However, the EIP and IDP are not used to generate M&E data in relation to these objectives.
The individual and early intervention plans are expected to be compiled upon finalization of
the enquiry at the children’s court to guide the social worker in the planning of services to be
rendered. However, due to the specialized structure of the agency, upon finalization, the file
is transferred to another unit, a process which is time consuming and often results in gaps in
service rendering when another social worker takes over the case to supervise and monitor
the placement. Moreover, due to the nature of challenges faced by child welfare agencies,
such as high staff turnover, bureaucratic pressures, low worker compliance with evaluation,
and high caseloads (Lawrence et al., 2013, p. 6), these plans may not be prioritized by social
workers especially because they are compiled manually which is a tedious process.
The CP is also compiled by a social worker once a child is placed in a CYCC. It has a strong
emphasis on family reunification and community-based care of children. The CP is hand-
written on a template by a social worker and checked for quality by a supervisor. It is then
filed by a social worker rendering reunification services and a copy is submitted to the
residential social worker in a CYCC where the child is committed. The CP serves as a tool to
guide the reunification process and review of set goals during case conferences held between
the reunification social worker, the residential social worker and the child on a quarterly
88
basis. The student found out that this plan is vital in speaking to objective 5 in the annual
business plan, namely “to place 45 children, who are either in residential care or in foster care
back with their biological families and to support and monitor these placements”.
Nonetheless, the CP is not utilised to generate M&E data – it serves only to facilitate service
delivery. Given that the CP is compiled in respect to children in CYCCs, there is a gap that
exists as there are no plans visibly documented for the reunification of children in foster care.
The student also analysed the daily diary (refer to Annexure J), which is a tool used to
supervise individual workers in terms of time allocated to different activities associated with
service delivery. It is more or less a log sheet of the daily activities for each day recorded
manually by each social worker. The daily diary is submitted to the supervisor at the end of
the month who checks to see whether it corresponds with the record of services rendered as
reflected in the SCS and on the statistical templates completed by each social worker for
internal purposes and submissions to the DSD. Afterwards, the daily diaries are submitted to
the manager who randomly checks them for accuracy and then they are filed. The daily diary
is used as a source of M&E information required on the forms completed for the DSD and for
internal statistical purposes.
As part of the documents used at the point of delivery to generate M&E data, the student
analysed the templates used to compile information required for internal statistics, first for
agency purposes and, second, for submission to the DSD. These templates are completed by
each social worker manually at the end of each month. The internal statistical information
form (refer to Annexure M) captures information that relates to inputs (such as caseload size
and the demography of clients) and outputs (such as services rendered, reports written).
These data are manually compiled by social workers, drawing on their paper files. However,
most of the data on this form is already or could be easily captured on the SCS, producing an
instant and more easily verifiable report of inputs and outputs.
The templates for generating statistical information required by the DSD (refer to Annexure
N) gather data that relates to inputs and outputs in terms of supportive services rendered to
children and their families. The templates serve as registers for clients (children and
biological parents) who received services for the month. They also provide information that
relates to demographic profile indicators and input indicators. By and large, they seek to
monitor performance indicators in terms of eight aspects, namely:
89
1. Family preservation: Number of family members participating in family preservation
programme provided by the funded NPO.
2. Family reunification: Number of family members participating in family reunification
programme by funded NPO.
3. Parenting programmes: Number of family members participating in parenting
programme by funded NPOs.
4. Successful family reintegration services: Number of family members reunited with
their families through services provided by funded NPO.
5. Crisis intervention: Number of families receiving crisis intervention services (social
work services).
6. Increase in foster care caseloads: Number of children placed in foster care by funded
NPO (new placements).
7. Services rendered: Number of children placed in foster care that received social work
services by funded NPO.
8. Adoption services: Number of children awaiting adoption in terms of assessment,
matching and placement by funded NPO.
As these performance indicators relate to the agency as a whole, each unit completes
information relevant to the services it renders. For instance, the Foster Care Unit completes
information related to the objectives of family reunification, family reintegration and services
rendered for children in foster care (items 2, 4 and 7). Information gathered using these
templates (internal monthly statistical form and the DSD’s monthly evidence based statistical
forms) reflect the community-based services and activities rendered by the agency. They
closely relate with the annual business plan as they demonstrate whether the unit is achieving
the expected objectives and outcomes. The student, however, realized that due to the fact that
the agency collects statistical information manually (not through a computer based
programme), the integrity of the statistical data generated by the agency might be
compromised as completion of forms is tedious and subject to errors. Furthermore, all of this
data could be extracted from the SCS, based on process notes of services rendered to clients,
provided the key data elements were included on the process note form.
90
4.3.3 Processes and Tools Used to Translate Data into M&E Indicators
Since M&E activities at the ‘low’ level of the system determine whether overall objectives
and goals will be achieved, M&E data gathered at this level by social workers is submitted to
the middle line management consisting of supervisors for quality check. Supervisors also
provide support to the social workers through individual supervision and meetings. Minutes
of supervision meetings are, however, not recorded and reported on, which is a gap in the
system. These meetings relate to quality of services and have a bearing on the improvement
of the quality of the M&E processes.
Supervisors physically check and compare information captured on the templates with
information captured on other tools/forms such as the daily diary and the SCS. The
information required by the DSD is checked only for quality and accuracy, and it is not
collated at this stage. Should there be errors or anomalies, the supervisor discusses these with
the social worker who makes necessary changes. Once this is done, the supervisor then
submits this information to the unit manager, who also physically checks for accuracy.
Anomalies are then rectified between the supervisor and the responsible social worker. Once
satisfied with the accuracy of the information, the manager then gathers all the documents
and collates the statistical data into one document with total figures for the entire Foster Care
Unit. This data (templates completed by respective social workers and the document created
by the manager) is then forwarded to the Finance Officer, since statistical data are tied to
funding and correspond with DSD’s way of checking for compliance. The Finance Officer
then collates the statistical data gathered from all specialized units into one document for the
whole organization and submits the statistics to the DSD.
There is collation of the internal monthly statistics whereby supervisors use a Microsoft
Excel Spreadsheet to merge data obtained from different social workers in their respective
teams (refer to Annexure P). The supervisor then submits the spreadsheet together with the
forms compiled by each individual social worker to the unit manager. The aggregated team
level data could, as with the social worker data, be compiled from the SCS provided the
necessary data elements are recorded on the process notes template and captured in the SCS.
The fact that supervisors and managers manually check for quality is concerning as this is not
the most effective method. It presents the need for the agency to effectively utilize the SCS
and also to develop standardized tools which can be used to measure data quality and assess
91
the strengths and weaknesses of the agency’s M&E system (including data collection,
compilation and reporting at all levels).
Once supervisors have submitted all this information from their teams, the unit manager
collates the internal statistical information into one form capturing the total statistical data for
the whole unit. This is done using the Microsoft Excel Spreadsheet similar to the one used by
supervisors. The manager also collates the data required by the DSD onto one form with total
figures for the whole unit. This is done manually as there is no computer-based programme or
electronic version of the DSD evidence based template on which to capture the statistics. The
manager submits all the statistical information to a designated staff member responsible for
handling them. It is not understood what happens thereafter, that is, whether the statistics are
analysed, how statistical information is used to report to the top management and
stakeholders.
Findings reveal that services, supervision and M&E processes at the ‘lower’ level and middle
line management do not speak to the performance indicators enunciated in the annual
business plan. The student also found out that most of the forms/templates are not electronic,
a situation which makes it difficult for the agency to ensure data integrity. It is possible for
errors made at the social worker level to be overlooked at the supervisory level since there are
no strict measures to check for quality. The systems used by supervisors to collate statistical
information are different from that of the manager, a situation which presents some form of
inadequacy and inconsistency. Data integrity may be also compromised at this level
considering that the unit manager manually enters data gathered by supervisors onto their
Microsoft Excel Spreadsheet when collating statistical data to come up with statistics for the
overall unit.
The system, being largely manual, with handwritten lists that are repeatedly collated as they
move from level to level, is very labour intensive and repetitive, as similar data is recorded on
different templates.
As alluded to earlier, the agency has recently adopted a web-based computer programme
entitled the Soweto Care System (SCS). As of 2012/04/10, the agency had commenced
capturing client information onto the SCS, following a training of staff by software
developers on how to use it. Resource shortages (computers, time and personnel) were
92
reported in the minutes dated 2012/06/11 as challenges deterring progress in the
implementation of the SCS. It was remarked in the minutes dated 2012/07/09 that the SCS
should be included in the procedure manuals as a monitoring and evaluation system being
piloted (Annual Report, 2012/2013, p. 6). However, the student could not obtain any
documents from the agency that relate to the SCS, which could have been a source of
information on how the agency intends to use the SCS as an M&E tool. This suggests to the
student that the agency has an M&E tool which, however, does not have important guidelines
on how M&E data will be handled in informing the agency about its performance. In spite of
this, it is highlighted in the minutes dated 2012/10/08 that the SCS would be customized to
produce statistical reports as required by the agency for internal purposes and for submission
to the DSD. It is striking from the minutes analysed to note that the Foster Care Unit seems to
have prioritized SCS more than other units in the agency, in that the foster care manager gave
more feedback and took more initiatives to come up with possible reports that could be
generated from the SCS as compared to the other specialised unit managers. The use of SCS
in the agency has therefore not been uniform among different units.
4.3.4 Management of Processes
In order to have a clear understanding of how the processes at the point of service delivery
and the middle management level are managed, the student analysed the minutes of
managers’ meetings. The student found out that the managers’ minutes principally report on
issues related to funding, finances, planned events, staffing, advocacy services, community-
based initiatives, administration and human resource management. The minutes do not
necessarily have value in terms of their usage for M&E but they provide information about
the agency’s priorities. The student found out that M&E is not a topical issue in the
managers’ meetings which suggests that managers are not managing M&E processes (which
may have an impact on the quality of M&E data) and are not using M&E results to improve
the programme. Both of these are M&E process gaps, suggesting that M&E is not adequately
integrated into the business of the programme. Minutes of managers’ meetings dated
2013/08/12 highlight that “monitoring and evaluation is key for DSD” and therefore
managers were encouraged to incorporate this aspect (M&E) into their business plans where
possible. This finding, that M&E is regarded as more of a DSD concern than an agency
concern, suggests to the student that the agency has not fully bought into the concept and
93
value of M&E. Consequently, M&E activities and utilisation have not been incorporated fully
into the planning and conducting of the agency’s business.
M&E training became an issue on the managers’ meeting agenda on 2012/10/08, where it
was highlighted that only managers and supervisors would be trained in M&E the following
year. Nothing was highlighted regarding the training of service delivery level workers in
M&E. By 2012/10/08, only one staff member from an outside facility had attended an M&E
course offered by the Regional Psychosocial Support Initiative (REPSSI). Eight M&E
training sessions were scheduled between June and December 2013 for managers and
supervisors to attend. The managers’ minutes dated 2013/03/11, emphasize that “M&E is
vital for funding proposals and it is important to get buy-in from managers.” This seems to
imply that M&E training is meant to enhance the ability of the agency to attract funding and
not necessarily to inform service delivery improvements. The above quotation also suggests
that there is not enough buy-in on M&E training from the managers, hence they were being
urged to buy into the M&E concept. This is further substantiated by the fact that there was a
waiting period of eight months before training could commence even though it was being
offered to the agency on a pro bono basis. Managers are noted to have indicated that they
“did not have time to attend training” at the point when the issue of M&E training was
introduced (Managers’ minutes, 2012/11/12) which further illuminates on the finding that
there is not equal commitment to M&E training among the management team.
4.3.5 M&E Reports of Indicators
To close the gap between the way the agency manages M&E processes and the ‘high’ level at
which the agency produces documents governing M&E processes, the student conducted an
analysis of the six-monthly progress reports, the annual report and statistics for internal and
external stakeholders (including DSD). These documents report on the programme goal and
objectives and reflect how they are aligned with the DSD guidelines for M&E. The following
findings, which emanated from the study of the aforementioned documents, are presented.
The six-monthly progress reports are the M&E outputs. They report on the M&E results that
the programme generates drawing on the administrative records and statistical data collated.
The following kinds of data are produced from the six-monthly reports:
94
Training and capacity building: This entails the training of staff members as well as
clients. Training of staff members encompasses orientation, in-service training, seminars
and workshops on various topics meant to build knowledge and improve skills. Training
of clients includes parenting skills and training of prospective foster parents.
Programme goals and objectives: This is an account of the input, activities (processes)
and outputs in respect of the six objectives highlighted above in relation to the annual
business plan. The six-monthly progress report outlines the achievements (outputs), the
challenges encountered and the plan of action devised to overcome those challenges in
order to achieve the objectives. The following example has been extracted from the April
- September 2012 progress report to illustrate how the agency presents its information.
Objective 4: To provide supportive and empowering services to 800 foster families to
enable them to provide suitable care for the children.
ACTIVITIES ACHIEVEMENTS CHALLENGES REMARKS/ PLAN
OF ACTION
- Counselling services
were provided as often
as was possible. The
required number of
home visits/ office
interviews were not
conducted in all cases.
- Services had been
rendered to 1059 foster
parents at the time of
the compilation of this
report
- Services were reactive
rather than proactive
- High caseloads
- Problems with the
Children’s Act
- Lack of driving skills
- Motivation of social
workers to obtain
driving licenses
- Advocacy for parity in
salaries
- Working of overtime
- Advocacy in respect to
the Children’s Act
The illustration above demonstrates how the agency is fixated on outputs since the report
does not encompass the outcomes and impacts being achieved by the agency. It also portrays
how the agency is able to report on quantitative outputs and it limitations in describing the
effectiveness of services rendered. The report states the challenges being experienced by the
agency in achieving its objectives and makes suggestions on how to overcome them.
Impact of the services: This section should conceptually report on the long-term effects of
service delivery on clients. Examples of ‘impacts’ reported in the April-September 2012
progress report include: “51 children were reached by means of group work”;
“networking with schools took place”; “family homes were available for 82 additional
children in need of alternative care”. These are output indicators, describing the quantity
of services rendered and beneficiaries reached, rather than the impact of these services on
the lives of the beneficiaries.
95
Improvement plan: This entails strategies that the agency will employ in order to improve
on service delivery and their overall ability to achieve the expected objectives. This
includes aspects related to staff retention and how service delivery will be improved (e.g.
rendering in-depth therapeutic services)
Monitoring and evaluation plan: No information was recorded under this section in all
the reports analysed. This suggests to the student that there is an inadequacy of M&E
capacity which results in data of low quality that do not provide for effective decision
making. It also suggests to the student that the M&E training which was offered may not
have resulted in M&E system strengthening and gaps still exist which could be covered
by M&E capacity building among programme implementers, supervisors and managers.
The student found that the template used for compiling the annual business plan is the same
template used by managers when compiling the six-monthly progress reports. The student
also found out that there was scant detail provided in the annual business plan and no
information at all in the progress reports under the heading ‘monitoring and evaluation plan’,
which suggests the need for M&E training. The student established that M&E components
(e.g. outputs, outcomes and impact) continue to get confused in the progress reports as in the
annual business plan which further substantiates the need for M&E training.
It was established through analysis of the statistics for internal and external stakeholders that
the information presented in the progress reports and the annual reports originate from
statistical information collected by social workers and collated for the unit as a whole. This
statistical information focused entirely on output indicators.
The agency’s annual report reports on the challenges being faced by the agency and strategies
employed to address the challenges; the extent of services and how each unit in the agency is
doing in terms of rendering the specialized services; new community-based initiatives; and a
breakdown of financial records. It is an all-encompassing description of the inputs, activities
and outputs. The annual report is reflective of how the agency is concentrating its efforts on
measuring outputs, whilst an insignificant amount of information is provided to report on
outcomes and impact of service delivery. This gap in the agency M&E system is realised
where it is reported that:
96
“A newly appointed researcher will undertake evaluative
research that should strengthen and guide our service and
advocacy activities and assist with the development of
monitoring and evaluation tools in order to measure impact.”
(Annual Report, 2011/2012, p. 5)
This perhaps reflects that agency is aware of its M&E shortcomings, hence they are
gravitating towards the measurement of impacts since it is a gap. It is, however, still not
understood how the agency will improve its M&E system in order to move from measuring
inputs, processes and outputs towards the measurement of outcomes and impacts.
4.4 Conclusion
Analysis at this phase indicated a number of findings that help describe the agency’s M&E
system. Primarily, the M&E system points to a lack of coherent strategic direction and
guidelines leading to poor M&E implementation that does not necessarily generate M&E
knowledge useful in improving the M&E system and service delivery. There is also a lack of
buy-in and equal commitment to M&E among the management team. ‘High’ level documents
do not reflect specific and concrete M&E plans. Resultantly, there is lack of budget and
expert staff dedicated to M&E. Staff at the service delivery level, responsible for generating
M&E data, are incapacitated to operate and participate in the M&E system. The M&E system
also reflects a fixation on inputs, processes and outputs, and neglect of outcomes. It is highly
manual and repetitive in nature resulting in the electronic M&E tool (SCS) adopted by the
agency being underutilized.
Given these findings, the student conducted interviews with the social workers, supervisors
and managers involved in the M&E system to have a better understanding of how the system
works in practice. The following phase therefore presents and discusses the findings obtained
during these interviews.
97
Chapter Five: Phase Two Findings: M&E in Practice
5.1 Introduction
Through the means of documentary analysis, the preceding chapter (Phase 1 findings)
investigated the M&E situation as it exists in the agency under study. The descriptive and
illustrative findings that were generated from the documentary analysis helped the student
understand why the agency’s M&E system is the way it is. Several recommendations on how
the agency’s M&E system could be improved were also made by the student. Nevertheless,
findings in Chapter 4 raised questions that needed further exploration and appreciation of
how the agency’s M&E system really works in practice. As such, interviews were conducted
with a sample of social workers, supervisors and managers involved in the M&E system. This
chapter therefore presents and discusses findings that stemmed from these interviews.
First, the biographic details of participants are presented using tabular illustrations. Such
visuals are followed by a brief description of the findings. The findings from the semi-
structured interviews are then presented and discussed in the form of themes and subthemes
in the sequence described below.
The student commences presentation of what emanated from the semi-structured interviews
by exploring findings that are related to the agency’s programme structure. Thereafter, a
discussion pertaining to the agency’s understanding of M&E, the developmental approach,
indicators, and how these concepts connect with community-based services for OVCs offered
by the agency follows. Subsequent to this, is a discussion of the findings on the agency’s
‘high’ level documents (policies, guidelines, business plans and indicators). Drawing from
the information obtained in the semi-structured interviews, the student explains how M&E
plans (as captured in the high level documents) are being implemented at the point of service
delivery by social work practitioners who are engaged in various M&E processes using a
variety of tools and templates to collect M&E information. Afterwards, a consideration of the
M&E processes conducted by supervisors and managers related to the processing, analysis
and use of M&E information in the agency is made. The findings obtained at this level are,
next, linked with the reporting structures of the agency coupled with a discussion on whether
information that is reported on resonates with the goal and objectives spelled out in the
98
business plans. Whilst presenting findings in this phase, the student highlights
recommendations that participants made for each particular theme and/or subtheme. A
discussion on recommendations relating to how M&E could be improved in the child welfare
field, and a conclusion thereof, closes this chapter.
5.2 Demographic Profile
The demographic profiles were constructed by drawing information from the participants on
their gender; age and racial group; years of experience in child protection services; years of
employment in the agency as social workers, supervisors or managers; years of experience
implementing the Children’s Act No. 38 of 2005; their involvement in the implementation of
the Child Care Act No. 74 of 1983; and from the type of services that they render. The
demographic profile of participants is presented in Table 5.1 below.
Table 5.1 Demographic Profile
Name Role Gender Age Race
SW1 Social worker Female 30 African
SW2 Social worker Female 30 African
SW3 Social worker Male 32 African
SW4 Social worker Male 28 African
SV1 Supervisor Male 31 African
SV2 Supervisor Female 64 White
M1 Manager Female 63 White
M2 Manager Female 41 African
Of the eight participants in the study, three were male and five were female. This occurrence
is a reflection of the dominance of women in the social work field. It was also noticeable that
females hold higher ranks in the agency (two managers and one supervisor) as compared to
the males in the agency. This observation is consistent with the notion of female dominance
in the social work field.
99
Table 5.2 Work Experience Profile
Name Role Years in
Child
Protection
Services
Years in
the
Agency
Implementation of
the new Children’s
Act (Years of
Experience)
Implementation of
the old Child Care
Act (Years of
Experience)
SW1 Social worker 6 4 4 2
SW2 Social worker 4 4 4 0
SW3 Social worker 5 4 4 1
SW4 Social worker 4 2 4 0
SV1 Supervisor 38 38 4 34
SV2 Supervisor 5 5 4 1
M1 Manager 30 30 4 26
M2 Manager 9 3 4 5
All eight participants have been involved in the implementation of the new Children’s Act
since it was effected four years ago and became the overall policy governing child welfare
practice. There is a relatively small number of participants who have significant experience in
the implementation of the old Child Care Act since the majority of the participants are young
adults who have 4-6 years of experience in the field. Supervisors and managers have more
years of experience in the child protection field and in the agency as well.
The information on the type of services that the eight participants provided reflects that the
majority provide foster care supervision and monitoring services as well as statutory services,
counselling and therapeutic support, supportive and empowerment programmes, and child
protection programmes. This is reflective of the sample which was drawn as the majority of
the participants were sampled from the Foster Care Unit which provides such services. These
findings also indicate that most participants provide core services of child protection across
the respective levels of child protection, namely statutory services, monitoring and
supervision of foster care placements, and family reunification services. Developmental
programmes are provided to a lesser extent.
100
5.3 Overview of Findings Emanating from the Semi-Structured
Interviews
In order to fulfil the second and the third objectives of this study, semi-structured interviews
were conducted with the view to generate findings that help clarify and evaluate how the
agency’s M&E system works in practice. The student begins this section by discussing the
overall programme structure of the agency but first presents below a tabulation of the themes
and subthemes that clearly maps out the structure of the presented findings:
Table 5.3 Overview of Semi-Structured Interview Findings
MAIN THEMES SUBTHEMES
Programme Structure of the Agency
Knowledge of the Principles Governing
Community-Based Services for OVCs Understanding of the meaning and purpose of
M&E
Understanding of the meaning and utility of
indicators
Understanding of the developmental approach
Community Based Services for OVCs
‘High’ Level Documents (Governance) Overall Policy on M&E
M&E guidelines (devised by DSD, the agency
&/or stakeholders)
Business plans
Indicators
M&E Processes at the Point of Service Delivery
Level M&E Planning
Social workers’ attitudes towards M&E
Measurement of community based services for
OVCs
Statistical information
Tools and templates
Soweto Care System as the agency’s M&E system
M&E Data Processing, Analysis & Use Role of supervisors in the M&E process (Data
collation)
Role of managers in the M&E process (M&E
planning)
M&E Reports Six-monthly progress reports
Annual report
Training Needs
Recommendations for Improving M&E in the
Child Welfare Field
The data is presented below according to the themes and sub-themes identified above.
101
5.4 Programme Structure of the Agency
In the previous phase, which discussed findings relating to the situation analysis, it was
highlighted that the agency has an established hierarchy (refer to Annexure H) where
instructions are given from managers to social workers through supervisors (top-down
approach). Accordingly, the M&E function is entrenched within the structures of the agency
creating an ‘institutional fit’ between the structure and the purpose of the agency (Ackron,
2013). One participant’s view subtly captures this conception:
In terms of how it’s [the unit] structured umm I would say that it’s my opinion that
it’s correctly structured and in terms of how services are rendered... we are not
rendering generic services and I believe that that’s the correct way to go. (Manager
1, #91)1
In other words, the participant believes that the agency is structured appropriately with
respect to the services being rendered and how they are rendered through specialised units.
Nonetheless, the student realised from both the documentary analysis and the semi-structured
interviews that this structure places excessive focus on service delivery at the expense of
M&E functions, which makes other participants sceptical as to whether the agency has a
defined M&E system:
The emphasis is more on processes than evaluation eeh monitoring and evaluation
because we are looking at what is the flow of work in the organisation but the impact
of that flow, how things would flow is not necessarily measured. Yah so I don’t think
there is emphasis on M&E. (Manager 2, #314)
I think at this juncture I will be forced to think again whether we have an M&E
system at all in this organisation... there are bits and pieces of an M&E system but we
don’t have this one holistic you know which is combined M&E system that we can
clearly point out and say “this is our M&E system”. (Supervisor 1, #226)
Moreover, the structure of the agency hinders active and full participation of those involved
in implementing the plans, a situation which stifles creativity and slows down ability of the
1 The text in brackets alongside each participant’s quotation includes the role and number of the participant who
was interviewed (corresponding with Tables 5.1 and 5.2 provided earlier) and the line number in the data set
from where the quotation was extracted.
102
agency to achieve set goals because of its emphasis on paperwork and following orders and
protocols (Weber, 1947). Each unit seems to have its own agenda and there seems to be little
or no cooperation between units to get the job done. According to Martin (2003, p. 17),
bureaucratic organisations often become an “iron cage of control” where red tape and
compartmentalisation of staff is the order of the day. Ad hoc meetings within such a
restrictive top-down structure (Martin, 2003) do not necessarily result in information (about
objectives, targets and indicators) filtering through to the practitioners who are implementing
the plans:
I think they need to strategize the involvement of staff because at the moment
everything just trickles down from the top management down to us. Of which I would
think it can start from the bottom which is us. They should consult us from the bottom
because we are the ones who are more involved in the field; we are more involved
with interactions with the clients. So I think they need to change and start from the
bottom which is us we give them information, we give them what we think we give
them our ideas and they should consider them also. (Social Worker 4, #566)
This is a huge organisation we don’t get involved in such kinds of things. And
somehow I don’t know have I ever seen any business plan here? Never! And I do not
know what is in those things. What I know is that I should be supervising social
workers and this is what social workers should be doing whether we are meeting
targets or not, I do not know. Why? Because I am not involved. I have never been
involved and I don’t think they want to involve me. (Supervisor 1, #554)
Yah I don’t know but this agency is quite disjointed you know in a way in that there
are people operating in their own teams aah management are there yes but the
directors etcetera are very remote aah they often feel as though they are not part of
anything you know. (Supervisor 2, #637)
Managers’ reports? I have never seen one in my entire life. Besides being told that I
am compiling my manager’s report and besides being told like in the manager’s
report there is this and that and that and that but what is in that report, it doesn’t get
you know disseminated back to us. (Supervisor 1, #635)
In light of these findings, social workers and supervisors feel that there is a strong need for
the agency to find ways of encouraging participation, such as a bottom-up approach. More
ideas could be generated and implemented through the active participation of practitioners
103
involved in the implementation of the plans. Communication flow between the practitioners
at the point of service delivery and the management that makes the plans could also be
improved so that there is mutual understanding of the plans, a situation which creates buy-in
and ownership of the programme for OVCs (World Bank, 2013).
5.5 Knowledge of the Principles that Govern Community-Based Services
for OVCs
Since the study seeks to explore how community-based services for OVCs are being
monitored and evaluated in the child welfare sector by and large, the knowledge of the
participants of key principles was explored, focusing first on their understanding of the term
monitoring and evaluation. Second, the participants’ understanding of the term ‘indicators’
was explored as indicators are building blocks of M&E (Imas & Rist, 2009). In view of the
fact that child protection services are provided from a developmental approach, the
practitioners’ characterization of the concept of social development was delved into to
ascertain whether it was directly aligned with the premises on which the developmental
approach is built. The following subthemes are, therefore, presented with direct quotes to
give voice to the participants’ views.
5.5.1 Understanding of the Meaning and Purpose of M&E
Monitoring and evaluation has a fundamental role to play in making conclusions about the
significance of a programme in terms of its design, usefulness and longevity (Rossi, Lipsey,
& Freeman, 2004). In light of this, an exploration of the participants’ understanding of the
term M&E was conducted. The participants reflected a fair understanding of the concept and
purpose of M&E:
My understanding of that is well it’s twofold: monitoring in terms of quantity and
should also be in terms of quality. Aah and in the end coming up with an evaluation of
how we are doing in terms of quantity and quality. In terms of the services that we
render. (Manager 1, #134)
I would see M&E as sort of, as a strategy which is being implemented by
organisations to ensure that they are effective in their service rendering to measure
104
the progress that they are doing as far as meeting their goals, their agenda, their
mandate and their mission statement is concerned. (Supervisor 1, #195)
I can just split it and say, I can say monitoring is when you do like an assessment of
maybe of a project or programme that you are doing maybe at each level of a project
or a programme you do an assessment... then evaluating is the assessing the project
as well, maybe at the end of a project or maybe at the end of a programme the
effectiveness and the impact that it is... that it has caused. (Social worker 2, #97)
As highlighted in the documentary analysis phase, only managers and supervisors had been
trained in M&E, but even social workers who have not been involved in that training
exhibited an adequate understanding of M&E. However, it could be realised from the
interviews that the participants tended to associate M&E with the collection of ‘statistics’.
M&E was linked to the responsibility that the agency has of being accountable to its funders
(mainly DSD), rather than the major role of M&E in making deductions about a programme’s
value and standard (Rossi et al., 2004), and in improving service delivery (Rabie & Cloete,
2011):
Monitoring would be your gathering of information through the services you are
rendering, your statistics umm your reports umm yah and then evaluation would be
evaluating statistics in order to plan... (Supervisor 2, #89)
Monitoring and evaluation. You know every time I hear about this term I think about
money. The only time we get asked about issues of M&E is when funders wants to
justify the how or wants us to be accountable as to how we spend the money that they
gave us. (Supervisor 1, #189)
Given that child welfare practice is donor-driven and impelled by state funding, this finding
is consistent with Noyoo’s (2005) contention that M&E improves accountability to key
stakeholders and provides a forceful platform for raising funds. Nonetheless, the agency
could strive to strike a balance between this role of M&E and that of improving programme
efficiency, responsiveness and relevance (UNICEF, 2009b). M&E training could help
demystify misconceptions of the purpose of M&E and equip practitioners in the agency to
effectively execute their M&E functions.
105
5.5.2 Understanding of the Meaning and Utility of Indicators
UNAIDS (2010, p. 77) highlights that indicators are variables that provide a “valid and
reliable” method of measuring programme performance. Thus, the interviews sought
participants’ understanding of the meaning and utility of indicators in the agency.
Participants’ responses of the question on indicators reflected a lack of understanding of what
indicators are. Asked the question “Are there any specific indicators of change that you track
in rendering community-based services to evaluate whether you are making a difference?”
the following responses were given:
Mmmmm I would say there is. In a sense that you can if you are able to set goals in
the beginning then it’s easier to find whether the services that you [are] providing are
effective or not, but if there are no goals you are just going there to reach the target
then there is no... (Social worker 3, #103)
Yah I think we are always aware of what the trends are within the adoption
community aah you can see from one year to the next how you are doing in terms of
placement of children whether there has been an increase in the number of adopters
who approached you whether there is a decrease. (Supervisor 2, #330)
Mmmm I think this one is a very tricky question but what I would think to be as an
indicator is eeh I will give an example of a child who is... who was removed from
foster care placement into an institution. So according to my understanding I would
think what would be an indicator in this case would be behaviour, will be the
behaviour of that child. (Social worker 4, #376)
Social work participants confused indicators with goals or trends. However, managers and
social workers were able to give satisfactory responses and they drew examples of indicators
from their business plans, which illuminate their pivotal role within the agency’s M&E
system:
Indicators I would say they are like guidelines; things that would make you see
whether you are getting to or you are doing what you should be doing. (Manager 2, #
211)
I will say indicators you know these are the things that we would look for that would
indicate whether we have achieved the certain objective. Like each objective should
106
have indicators ok to say for you to say you have actually achieved this, this should be
visible, right. (Supervisor 1, #515)
Findings for this theme suggest the need for the agency to pull resources towards training its
staff in M&E and on the developmental approach (as will be highlighted below), which could
result in practitioners appreciating more their role in the provision of M&E data. The
following section will consider findings obtained with respect to understanding of the
developmental approach.
5.5.3 Understanding of the Developmental Approach
Since the provision of community-based services for OVCs is governed by the principles,
themes and practices of the developmental child welfare approach (Patel, 2005), the
participants’ understanding of the developmental approach, generally, was explored. There
was a variety of participant responses to the interview question: South Africa has adopted a
developmental approach to social work and social welfare. What is your understanding of the
term ‘developmental’? Participants found it difficult to explicitly define the developmental
approach and also to clearly locate the work that they do within the approach:
Well, developmental I see it as umm as growth or making a progress in the services
that are being rendered we need to measure that it is... there is... there is an impact
that is going on, in terms of the services that are rendered to... to the community.
(Social worker 2, #68)
For me I will say developmental it means a way of building the personal skills of a
person. (Social worker 3, #58)
Mmmm my understanding of the term developmental is that eeh for something to be
developmental it’s... eeh it has to eeh... it has to be sustainable and offer services
which are sustainable, which are not remedial which are sustainable, which can last,
which can change poverty or change circumstances. (Social worker 4, #80)
I think it’s looking at eeeh how do you enhance what people already have in terms
of... you look at the people’s strength you don’t just go as a social worker then oooh
people need your help and you are there to solve all their problems but you need to
look at what do they have and what can you do with what they have to enhance their
life status. (Manager 2, #184)
107
This finding is concerning because social workers are expected to deliver community-based
services for OVCs from a developmental perspective as regulated by the new Children’s Act.
Managers who are directly involved in programming community-based services for OVCs in
the agency are also expected to come up with plans that reflect developmental activities,
values and principles (Smart, 2003). In-house trainings and information sharing sessions
could bridge this gap in knowledge and demonstrate how the developmental approach can be
contextualized within child protection given that some participants elaborated the
developmental perspective satisfactorily:
So this is post apartheid era approach which was heavily influenced by the White
Paper. There are some gurus in the field; people like Patel, people like Lombard; they
would argue to say you know this developmental approach it was adopted in order to
reverse the injustices of past. It was addressed in order to ensure that SA moved
forward in harmony and in development, you know, and it was also adopted after
realising that the previous approaches to development had failed to bear any fruits...
The developmental approach is an approach which seeks to marry social development
with economic development; to harmonise the two. Ok, it is somehow a materialistic
approach to poverty. The developmental approach is also very big on the issues of
participation. It is somehow a strengths-based and a rights-based approach to service
delivery, which is what we do here. (Supervisor 1, #125)
Well, I think the developmental approach came about with the period of transition in
the South African government in 1994 with the White Paper and it looked at
empowering people to take responsibility for their lives and connecting them to
resources in their communities; trying to prevent children from being removed from
their families and developing capacities and in people. Yah, focusing on early
intervention and prevention really more than anything. (Supervisor 2, #64)
The fact that such a detailed understanding of the approach emanated from participants who
are supervisors in the agency is not a mere coincidence. Noticeably, it is a reflection of the
supervisors’ experience in the field, their eagerness to learn given that their role is to provide
guidance to social workers, and the expectation of the agency on them to ensure quality in
service provision.
108
5.6 Community-Based Services for OVCs
It is crucial for child welfare agencies to provide wide-ranging, high quality services that
promote, restore, strengthen and maintain the maximum level of social functioning for
children and their families within their homes or community (NAP 2009-2012, p. 9). As
highlighted in the documentary analysis phase, the agency strives to uphold this principle by
providing various community-based services for OVCs through specialised units. The agency
prides itself in rendering extensive child protection programmes and services, which are
deemed to be of a high standard, not only in Johannesburg but across South Africa. This
culture of pride emanates, perhaps, from the fact that the agency has been in existence for
over 100 years and that throughout the years, the agency has established programmes that
possibly respond to child welfare issues in a more effective fashion as compared to other
agencies:
I think we have a very high standard of practice and we are kind of the model agency
within the community or in South Africa, really, in terms of setting goals and
standards. And other agencies would ask for guidance when they are developing their
own programmes. (Supervisor 2, #113)
Nonetheless, the agency could still develop programmes that specifically deal with poverty
reduction, economic development, home and community-based care, as well as supportive
and empowerment programmes, which were not ticked by the majority of participants on the
semi-structured interview schedule. The current situation is that clients are either referred to
other service providers for such services or they are provided with piecemeal services that
have been instituted to cover this gap:
Eeh the programme that we have that relates to poverty reduction I would say the
foster care programme on its own it relates to issues of poverty reduction even though
you would not clearly see it. It is not clearly visible that it is a poverty reduction
programme... Foster care grant is money and that serves as to contribute towards the
household income of that particular family. So I would want to view our programme
as a poverty reduction programme in some way... Eeeh even if you look at the
programmes that we render to the biological mothers. We refer most of them to skills
training courses with the view that they can get employed. (Supervisor 1, #95)
109
I would think eeh the foster care system is aimed at poverty reduction I think so but
I’m not sure exactly if we have specific services which are aimed at poverty reduction
and development of our foster care children and foster parents. (Social worker 4,
#71)
Through the interviews, the student realised that the agency is determined to expand its
services and cover gaps by introducing new programmes, goals and objectives. However,
challenges in securing funding (Lawrence et al., 2013) often get in the way and deter the
agency from taking such initiatives:
You would include those [new objectives] in your business plan but what I am saying
is that that does not necessarily mean that that would equate to your subsidy being
increased. It might ukhuti [be that] ooh ok you are doing well, keep on doing it with
the same kind of support that you have been getting before. (Manager 2, #573)
Despite this challenge which is prevalent across the child welfare sector (Lawrence, 2013),
the agency has managed to establish numerous programmes and services for OVCs and it has
enjoyed longevity. This speaks well about the resourcefulness of the agency.
Having explored the adequacy and comprehensiveness of the community-based services for
OVCs offered by the agency, and having appreciated the agency’s inventiveness amidst the
challenges in the child welfare field, the following section delves into the ‘high’ level
documents. A discussion of the ‘high’ level documents explains the policies, practices and
principles that are guiding M&E processes in the agency.
5.7 ‘High’ Level Documents
Under this theme, the student will explore findings that relate to the ‘high’ level documents
such as the overall policy on M&E that the agency has, M&E guidelines as devised by the
agency or set by the DSD, business plans, as well as indicators that the agency is tracking.
These documents are key in providing strategic direction to the agency.
110
5.7.1 Overall Policy on M&E
During the first phase of data collection, the student highlighted that no documents could be
obtained from the agency that relate to the overall M&E policy of community-based services
for OVCs. During this phase, the interviews verified that agency does not have clearly written
down policy on how community-based services for OVCs should be monitored and
evaluated. In light of the fact that there is no overall national policy and indicators to monitor
the implementation of the developmental approach in the welfare field (Patel et al., 2011) this
agency situation is a microcosm of the child welfare field more broadly. Rather, the agency
has tailored solutions. It is guided, by and large, by the Children’s Act No. 38 of 2005, which
is the overall policy for child welfare practice. This Act guides how child protection services
should be rendered from a developmental perspective (Sibanda, 2014), but does not
necessarily explain how services should be monitored and evaluated. Additionally, the
agency is restricted by the demands of the DSD who is the main funder. DSD, however, also
does not clearly stipulate how services should be monitored and evaluated. Rather, there is
insistence on specific information that the agency should report on monthly through
‘statistics’ and also through the six-monthly progress reports:
I think basically the Department policies would be covered in the business plans,
progress reports ummm. The agency policies are basically supportive policies for
staff not actually ahh evaluative policies in terms of what you should do, how you
should render a therapeutic interview for example or good quality of service. I think
the agency takes evaluations from what we provide elsewhere to other people outside
of the agency. (Manager 1, #248)
I would say that we are heavily ahh structured by the requirements of the Department
of Social Development and it terms of what we are funded for and our services have
to fit those objectives. (Manager 1, #164)
No there isn’t [agency policy] but there is an agency statistics form but it’s not... it’s
not in any form of a policy in terms of how services should be evaluated. (Manager 2,
#139)
We are also guided by... not guided by but restricted by the Children’s Act. And what
is required. I mean we can’t go to court with a report that you decide for yourself as a
social worker as to what you would put in there. (Manager 1, #168)
111
Ok, well the organisation has got statistics and the statistics that the organisation has
to give at the end of each month and that covers a wide range of services that we offer
and that does not necessarily go anywhere except for the organisation. And then we
have got the government department – Department of Social Development – that
prescribes what they want to know about what we do. So they would also ask us and
give us the points that we need to cover and you report on that. But also every six
months we have to give them a progress report that talks to the work that we have set
out in our business plan that we will do and we would look at how far are we in
achieving those objectives that we have set out. (Manager 2, #104)
5.7.2 M&E Guidelines
As is the case with the agency’s overall policy on M&E, there are no explicit or detailed
guidelines crafted by the agency or by DSD to govern how services should be monitored and
evaluated. As highlighted above, the agency uses ‘statistics’ that are generated for internal
and external purposes as M&E information, and also reports to the DSD through the six-
monthly progress reports. Other stakeholders (e.g. donors) do not necessarily prescribe M&E
practises to the agency, even though there may be a reporting structure for the agency to
account how funds were used:
Agency policies I would say to you less so because the Department’s policies and the
Children’s Act take priority and our policies maybe just made to fit into [those
already existing policies by the Department and the Children’s Act, which is the
overall policy]. (Manager 1, #171)
No, not really I mean if you look at funders who, for example, fund eeh food or items
of clothing or stationery. We requested from them what we need but no not really that
any other stakeholders can tell us what to do because there cannot be a clash with the
main funder which is the Department. (Manager 1, #185)
Some would fund you and want to know how you have used the money because
sometimes you would say this is what you are going to do so you report back in terms
of what you have asked for as to how [or] whether you have managed to do that. And
some would just give blindly and it’s up to you to report to them what you have done.
(Manager 2, #146)
112
5.7.3 Business Plans
Since the documentary analysis raised questions about the usefulness of the business plans in
guiding M&E processes and improving programme performance in the agency, this aspect
was further explored in this phase. It was realised that the business plan is a very structured
and restrictive document, which is important in justifying funding. Since incorporation of
new programmes in the business plans does not always amount to a significant increase in
funding from the main funder, objectives in the business plans hardly change from year to
year. Managers formulate and prioritise objectives by considering the composition of their
staff and the specialised services that they render as a unit, and then make a decision as to
what objectives to set, what indicators to track, as well as the outcomes to expect. As such,
the business plan is being used, more or less, as an administrative tool, rather than an
instrument to inform service delivery improvement:
Alright, it’s very structured, there is not much room for expansion, so if we decided to
do something like have an objective for economic empowerment, for example, I’m not
sure that the social workers will have the time for it because of what we have to do is
so structured you know. It really would take a lot more aah time than social workers
have to do anything else. (Manager 1, #484)
With the business plan because our services don’t change much, in terms of what we
do year in year out, it more or less remains the same. (Manager 2, #548)
Business plan yes is to get more resources but practically, because the business plan
is to the government department who does not give you the full amount. They give you
a subsidy so the challenge would then [be] how do you then bridge the gap in terms of
what would be lacking... Yah, so it becomes a bit tricky in terms of adding more things
because sometimes you think we actually need to do this but this would need more
resources (Manager 2, #553)
See, you look at what is it that you do. You think about your team and you think this is
what you do and then you [set] all those objectives. (Manager 2, #584)
The business plan is also a platform for managers to construct comprehensive M&E plans,
but due to lack of expertise in devising M&E plans, managers often inadequately capture
such plans. This leaves gaps in the agency’s M&E system, in that there is insufficient
113
information to provide strategic direction and guidance for managers to execute their M&E
duties effectively:
I wouldn’t say it [the M&E plan] is very much comprehensive. I think it [the M&E
plan] could get better. I think it could be structured a little bit differently because
number one, the organisation is very big and the each unit, even though interlinked,
they do very different things and then in terms of the monitoring and evaluation, it’s
kind of difficult to monitor from an agency point of view but from a departmental
point of view, I think it would be looking at the things that we had set out and then the
indicators that were put out to date; are we achieving what we said we would
achieve? (Manager 2, #247)
5.7.4 Indicators
Earlier on, in Section 5.5.2, a presentation of the findings related to the participants’
understanding of the meaning and utility of indicators was made. This section will,
conversely, present findings on whether the agency has overall indicators which are relevant,
clear and concise (Scott & Marshall, 2005) to monitor the implementation of the
developmental approach (Patel et al., 2011) and to demonstrate how well the agency is doing
in terms of its OVC programming (UNAIDS, 2010). The student explored with the managers
the six indicators that the agency is reportedly tracking (namely, distribution of services,
quality of services, quantity of services, time frame indicators, accessibility, and issue and
response specific indicators) and how they inform the overall M&E system of the agency.
Whilst managers identified with the cluster and sequence of the agency’s M&E system (refer
to Annexure K), it was the finding of this study that the above mentioned six indicators are
not clearly written down anywhere:
No, they are not [recorded]... in the business plan, it does not reflect, for instance,
how do you make your services accessible. Quality of service - it’s more implied. You
would get from what you read that ok, this is good quality service but in terms of how
it’s measured I don’t think it’s very clear. Quantity of service yes because it’s
numbers everywhere. Distribution of services... yes because we work according to
areas so, yes, it would fall under that. Issue and response specific... yes because it’s
all in your business plan where you talk about what is it that you will be looking at
114
because then we talk about, for instance, abandoned babies this is what I am gonna
do about them so that is issue and response. Time frame... well, I would say time
frame would be annual because it’s an annual thing but the nitty gritty of like I was
saying [that] how we measure [that] from the door to the next door how long does it
take for the one child. Those nitty gritties are not specified anywhere. (Manager 2,
#766)
The explanation above reflects that these six indicators are implicit as opposed to being
clearly stated and directly written down. Instead, the agency has various performance
indicators, which are numeric targets of what the agency wishes to achieve as relating to the
objectives set in the business plan compiled per unit:
As an organisation we do not have, I am not aware of any. Maybe those indicators
you can set them yourself as a social worker to say this is what I have done but I want
to track on what is happening. So you set your own indicators and you make your own
follow up, and you do your own evaluation and your monitoring tool on your own as
an individual but as an organisation I’m not so sure if they are there. (Social worker
4, #289)
There is a whole lot of them [indicators]. So mainly they refer to numbers aah so I
don’t know if you want me to tell you what they are... but they mainly refer to the
numbers not the quality. (Manager 1, #253)
An additional finding is that the agency has objectives for which they do not have the tools to
measure the effect of services rendered. Moreover, indicators set for such objectives, do not
necessarily gather information which relates to the components being measured. For
example, in the 2012-2013 Business Plan for the Foster Care Unit, objective 3 is expressed as
follows:
3. To render therapeutic group services and to build up self-esteem, confidence and self-
worth in 300 children placed in foster care through using this method;
The terms self-esteem, confidence and self-worth are not clearly defined and the numerical
indicators set (e.g. number of children at year end showing improved levels of independent
living skills as assessed by social workers; number of children at end of school year showing
goal-directed progress as assessed by the social workers) do not necessarily measure these
115
aspects and they do not speak to the quality and effectiveness of services rendered in
addressing issues of self-esteem, confidence and self-worth. The objective is qualitative in
nature and the agency does not have capacity, in terms of tools, to measure aspects of self-
esteem, confidence and self-worth. It is not clear from the set indicators how the Foster Care
Unit would know if they have built self-esteem, for example. There is, therefore,
misalignment between the objective and the indicators being tracked:
Like what I indicated, we do not do monitoring and evaluation services but eeh the
things that we track, to me, I feel that they do not speak directly to the type, quality
and nature of services that we offer to OVCs. The things that we track are things like:
are the orders still valid? How many orders have lapsed? Out of those lapsed orders,
how many did we reopen? Eeh you know it’s an indicator which says nothing about
the type of services. (Supervisor 1, #527)
As is highlighted in Section 5.5.2 with regards to the agency’s understanding of the concept
of indicators, there is an unsatisfactory appreciation of the meaning and purpose of indicators
among participants. This finding, coupled with the conception that indicators set at the
‘highest’ level of the agency’s M&E system are not effectively communicated to practitioners
implementing the plans, is a concern. Social workers and supervisors have had to devise their
own measures to monitor and evaluate services being delivered at the ‘lowest’ level of the
agency’s M&E system:
As an organisation we do not have [indicators]. I am not aware of any. Maybe those
indicators you can set them yourself as a social worker... and you make your own
follow up, and you do your own evaluation and your monitoring tool on your own.
(Social worker, #289)
So my own stats that I do personally that are not for any other person other than
myself umm are always relevant to me... I do report for myself at the end of the year
not the financial year but the end of the standard year – is that what you call it?
January to December – really just to analyse for myself. (Supervisor 2, #268)
Given this scenario, one participant succinctly captured that the agency has no clear policy
and guidelines to coherently and strategically direct how services could be improved:
I can say part of our developmental approach should be to develop better M&E
policies and guidelines (Manager 1, #231)
116
The agency could invest resources towards developing clear M&E policies and could also
allocate proper budgets towards M&E (USAID, 2010), since policy is what gives the agency
authorization to make specific decisions (Rabie, 2011).
This section has considered the sufficiency of the policies and guidelines that govern how
services should be monitored and evaluated in the agency. An appraisal of the standards and
different documents that encapsulate M&E procedures within the agency has also been
conducted. In light of this, the following theme illuminates on the M&E processes that occur
at the point of service delivery within the agency clarifying how the discussed M&E policies,
principles and practices help provide guidance for practitioners.
5.8 M&E Processes at the Point of Service Delivery Level
The ‘high’ level documents discussed in Section 5.7 above constitute objectives, indicators
and plans, which are essential in guiding how services should be rendered, monitored and
evaluated. They also assist the agency in determining which information to collect with
respect to the set objectives and indicators. It is therefore consistent and practical for
practitioners directly involved in implementing the plans to be privy to the information
contained in the ‘high’ level documents. Social workers and supervisors determine the quality
of services delivered, and they are key in the collection of relevant M&E information
required by management. Alas, it was established through the interviews conducted in this
phase that there are various shortcomings related to the involvement of practitioners in M&E
planning. This situation creates negative attitudes in social workers towards M&E and the
situation is further aggravated by the disproportionate collection of information and
measurement of services for OVCs. Various tools and templates used by the agency,
including the Soweto Care System, also create gaps in the agency’s M&E system. The
following is a discussion of the findings related to each of the subthemes.
5.8.1 M&E Planning
Due to the bureaucratic nature of the agency and the fact that the M&E function is entrenched
in the core business of the agency (Rabie, 2011), ad hoc meetings do not necessarily filter
down information concerning the set goals and objectives from the management to social
117
workers. Participants highlighted that they are hardly involved in M&E planning, staff
meetings or information sharing sessions about the contents of the business plans. Rather,
information in the business plans is kept confidential and M&E planning is considered the
preserve of managers:
No. I’m not informed of the M&E planning. I’m also not informed in the
establishment about the established goals and objectives. (Social worker 4, #419)
Only rarely when it comes to the business plan ummm then I’m included in terms of
giving statistics and also discussing realistically what our goals should be.
(Supervisor 2, #408)
This finding is concerning in that social workers in the agency seem to be implementing the
plans and tracking indicators of which they are not aware. It is therefore difficult for the
agency to get buy-in from the social workers and supervisors since they do not identify with
the goal and objectives and are not being involved in determining realistic objectives to
follow (Stufflebeam & Shinkfield, 2007). Participants feel like they have a peripheral role
despite the fact that they generate key information required by the agency:
I want to think it’s more of managerial who are involved in the M&E planning and
establishment of the overall goal... I think they should also start involving us because
we are not even involved in any of that in those aspects (Social worker 2, #223)
I can say I’m not involved in this process but I am part and parcel of the generation of
data for statistics because the information that this management use is generated by
us as social workers, we provide them with the data and they analyse the data and use
it for, I don’t know whether for monitoring or evaluation. So my role is a peripheral
role. I am not involved in the [M&E] planning. (Social worker 4, #299)
The agency could generate thorough information and create comprehensive M&E plans by
implementing a participatory M&E approach (World Bank, 2004). The agency requires
system-wide buy-in of social workers and supervisors in order to improving service delivery
and promote innovation when making plans for M&E (Stufflebeam & Shinkfield, 2007).
118
5.8.2 Social Workers’ Attitudes Towards M&E
Given the situation that practitioners, who are directly involved in the implementation of
plans set at the managerial level, are not involved in the planning, there is a lack of
understanding as to why certain M&E processes have to be followed and why particular
information needs to be collected. Resultantly, practitioners at the ‘lower’ level of the
agency’s M&E structure have developed negative attitudes towards M&E. Such negative
attitudes also stem from the fact that the process of collecting required M&E information
(statistics) is manual, tedious and repetitive. Data collected is numerical and participants
hardly understand the purpose and adequacy of such data. They associate the collection of
‘statistics’ with justification of funding and not necessarily a means of assessing the quality
of services they have rendered and what effect it has had on the clients, which is more
fulfilling to them:
Like what I said you know this compilation of stats it turns us valuable social workers
into glorified admin and clerical personnel, which I feel, you know, we do not have to
be engaging with papers. We should be engaging with people; changing lives, making
a difference out there. (Supervisor 1, #372)
Yah, the statistics thing is a laborious experience, time consuming and it requires
much of your mental effort especially in terms of meticulously capturing the
information. Aha, so most of the times we make a lot of mistakes in capturing the
information so we have to revisit, do corrections and do all those things. (Social
worker 1, #221)
Well, I think it’s just a boring process because more like just dealing with the
numbers of people that you see. But we don’t measure any quality. If really we have
rendered an effective service, we don’t. (Social worker 2, 163)
I think there is no buy-in. I don’t think that the social workers it’s a task that they
enjoy doing. Ummm they don’t see the relevance of it. As I have said, they don’t know
what happens to it, they don’t get feedback on it, so it’s just an admin task that has
needs to be done and once they are done with it, “thank goodness it’s over” and they
can hand it on and that’s it. (Supervisor 2, #244)
Eeh I would say this process is very tiresome, it’s manual; sometimes you find
yourself sitting the whole day trying to compile the data which will lead sometimes
119
into just copying and pasting previous statistics which you would have done then. It
would be more of repetitive. (Social worker 4, #287)
This finding is concerning, as negative attitudes, more often than not, lead to data integrity
being compromised. Participation of social workers in the determination of M&E information
to collect, fosters development and empowerment, creates shared understanding of the
programme, and enhances programme ownership (Rabie & Cloete, 2011). The agency could
benefit more from an adaptive, action-oriented strategy which builds stakeholder capacity and
facilitates collaboration and shared decision making for increased utilization of evaluation
results (Imas & Rist, 2009).
5.8.3 Measurement of Community-Based Services for OVCs
Since the purpose of this phase was to gain clarity on how the M&E system really works in
practice, the student inquired, through interviews, how the agency measures the effect of its
services. It was clarified through the interviews that the agency focuses heavily on
quantitative output indicators due to limitations posed by the demands of the DSD and the
fact that the agency does not have a dedicated M&E unit, which can guide the process and
devise tools to measure qualitative aspects of service provision. The following question was
asked to one of the participants: “Is it because you measure quantity because of the limitation
of the stipulations from the Department of Social Development through the business plan or
is it because the agency has not come up yet with tools to evaluate in terms of quality?” The
following response was given:
I think I would say it’s a little bit of both. From the Department of Social
Development’s side they are very geared to numbers: how many of this and how many
of that and how many of the next thing will then tell you how you are doing. From the
agency side umm we have been given very old out of date tools. However you have to
have people’s permission to use them and we wouldn’t even know in which part of the
world these people who drew them would find themselves and I don’t think they are
relevant for today. (Manager 1, #305)
Other participants highlighted the difficulties that lie in measuring the quality of services
rendered by the agency and attributed that difficulty to the complexity of the types of services
120
rendered; the fact that they (social workers and supervisors) are not involved in M&E
planning; poorly defined indicators; and a lack of specific developmental aspects to measure:
Our services have got many faces it is a multi faceted so it’s very difficult to measure
any progress in terms of service delivery. So the M&E systems that we have tend to
use the quantitative tools. They tend to use numbers and what we do here the
outcomes are mostly qualitative in nature. (Supervisor 1, #214)
We do not really measure the quality of services which we are rendering. What we
mainly do with our statistics at my organisation, is we concentrate more on the
quantity... we do not measure the quality because when it comes to quality my
understanding of quality is eeh are your services proper, are they suitable, are they
really serving the purposes that you intended, that you intend to. So I think we don’t
really measure the quality of the services. (Social worker 4, #161)
In order to have a better appreciation of the reasons why the agency is heavily focused on
quantitative output indicators, an inquiry into the statistical information collected by the
agency for both internal and external purposes was conducted. Moreover, the tools and
templates used to collect M&E information were explored. A discussion on these aspects
ensues below.
5.8.4 Statistical Information
As highlighted earlier in this section, ‘statistics’ play an important role in the agency’s M&E
system. In fact, StatsSA (2010) contends that statistics are indispensible in planning, resource
allocation, management control and determining reliability and validity in phenomena being
measured or analyzed. Social workers generate and collect various M&E information
(statistics), which constitutes output indicators. However, the agency’s M&E system
overemphasizes the importance of ‘statistics’ such that attention of practitioners seems to be
placed more on chasing figures than on improving service delivery, which is the major role of
M&E (Rabie, 2011). ‘Statistics’ are important for the agency as a means of ensuring
compliance, accountability and securing funding (Noyoo, 2005), rather than improving
programme performance:
121
The monthly statistics are part of the compliance that we have to... like we sign the
service level agreement which says we will do this they will give us so much but also
we will give them monthly statistics and the six-monthly progress reports and that. If
we don’t submit the statistics, then we don’t get the money. (Manager 2, #679)
It’s just a funding strategy to keep the money coming to say of course we are doing
something we have reached how many OVCs. (Supervisor 1, #412)
You know the experience about the collection of M&E [information] I somehow feel
you know it’s too much. It’s too frustrating... I feel like at times it is
counterproductive... The time that we spend compiling these statistics as required by
the agency and as required by the government could have been used to render
services to our OVCs. (Supervisor 1, #362)
Findings also reveal that the statistical information generated internally for internal and
external purposes does not necessarily translate into strategic measures to improve service
delivery. Participants feel that information gathered is inadequate, unreliable and inefficient
to inform service delivery improvement:
I can’t say it is adequate. It’s just numbers. It does not say anything about my
caseload. It does not say anything about my services. (Supervisor 1, #408)
The statistics that we are collecting on a monthly basis I think [they are] enough to
tell us like how many clients we render services to but then [they do] not actually
address the developmental needs of the clients. So I feel there is need for us to collect
additional information to say what are the changing trends out there and what is it
that our clients need? What more services could we put in place? How could we
reach a certain number of people that we have never reached before? (Supervisor 1,
#445)
I’m also questioning the reliability of the stats that we are actually given. (Supervisor
2, #224)
The agency has profound reliance on quantitative information. This numerical information
presents a limitation to the agency in that there is no wide range of variables to consider (such
as qualitative data) when examining data. This implies that the agency has to conduct
complex quantitative data analysis (Stufflebeam & Shinkfield, 2007) in order to benefit from
the information gathered. Seeing that the agency does not have capacity to analyze
quantitative data, no analysis is therefore conducted of the data collected. The collection of
122
quantitative data presents a scenario which is politically volatile since quantitative data can
be used to identify responsibility for failure to meet targets. This can create unhealthy
competition, unrest and cheating in order to ensure positive evaluation reports (Stufflebeam
& Shinkfield, 2007):
It’s a little bit moderate because sometimes you tend to reach the quantity number of
people that is needed for you to collect information from and at the end of the day you
end up not rendering the quality services to the people that you are supposed to
because you only doing it you tend to do it for stats as well even though you are
supposed to do it for the good service. (Social worker 3, #144)
The system will tell you: how many home visits did you conduct? How many
telephonic interviews did you do? What did you do? And stuff like that. Eeh which I
feel the system should not generate such kind of reports. The time that can be spent by
one social worker to see ten clients another one can spend the same time you know
working very hard on one particular case which is challenging which has got a crisis
and the system does not capture such kind of information to say these cases are
different since we will only say this one saw ten, this one saw one, which means this
one is not productive this one is lazy; this one is industrious, this one is hard working.
(Supervisor 1, #687)
5.8.5 Tools and Templates
Social workers use various tools and templates to plan their work and also to collect the
statistical information discussed above. Rabie (2011) contends that proper design of data
collection instruments is essential for reaching reliable and valid conclusions. The relevance,
usefulness and adequacy of the agency tools and templates were issues that were also
explored during the interviews. The agency uses numerous planning tools such as the daily
diary, Individual Development Plan (IDP), Care Plan (CP), Care Plan Review (CPR),
Reunification Services Plan (RSP) and the Case Supervision Plan (CSP). Whilst supervisors
managed to identify and articulate well the use of each of those tools, social workers could
mainly identify with the daily diary and process notes. Social workers hardly make use of the
planning tools (such as IDPs, CPs, CRPs and CSPs) perhaps due to the fact that they are
completed manually and that there is no uniformity across the agency. Pressure exerted on
123
social workers by the high caseloads, time constraints and generally a lack of will could also
be contributing factors (Lawrence et al., 2014):
There is also IDP. I have done it once at my current organisation. I have done it once
after it was requested by an organisation where I was placing a child. (Social worker
4, #205)
It is, however, important to note that the planning tools used by the agency, such as the IDP,
capture information which is useful in planning for service delivery by social workers and
reporting towards the ‘statistics’ required by the agency for internal and external purposes:
I think they are more than adequate if completed properly. (Manager 1, #376)
Yah really, they are they are really helpful in the sense that as we those templates they
try to make sure that we document each and every action that we do. Yeah so by
documenting every information or that particular information or service that we have
done, it makes easier when we are doing monthly stats to so that we recall everything
that we have done. (Social worker 1, #192)
They are a handful at the moment eeh at the moment I feel it’s too much of tools and I
think umm they actually capture the gist of the matter. (Supervisor 1, #323)
During the interviews, the student also explored the various tools and templates that are used
by the agency to collect M&E information (statistics). These tools and templates have either
been designed by the agency in times past or they were prescribed by the DSD. The process
of completing these tools and templates is equally tedious, repetitive and prone to errors since
they are manually completed. The various tools and templates that exist in the agency elicit
the same information, which is quantitative in nature. Participants questioned the relevance of
the tools and highlighted that they are not involved in the design of the tools and templates
and hardly get training on how to use them. Moreover, the tools and templates tend to be
technical and leave room for varying interpretations, which compromises the quality of data
generated using those tools:
Neither was I involved in the drafting of the statistical templates. Where did they
originate from? I don’t know... I was here I was not involved. Nobody called me to
say we are drafting a new statistical form. (Supervisor 1, #571)
124
I think our statistics needs to be computerized so that it does not take a lot of time to
compile them so that it doesn’t take a lot of time and just force us to make a repetition
of previous data which we would have compiled. (Social worker 4, #536)
To me it’s just compilation of stats. And remember I said it’s one stats; what you get
asked in one set of stats, you get asked in the next set of stats so to me it’s [repetitive].
(Supervisor 1, #428)
I think that they [data collection tools] leave room for interpretation which can be
bad because then how one interprets it and the other may differ and you find out that
therefore information is mixed because you feel that things might come into this and
yet they need to be going into a different section. (Manager 2, #346)
You know like, even the statistics that we do, we do about four sets of statistics
covering almost the same things. Can’t somebody just combine them and make this
one simple thing? And most of those tools they are very technical you know. You need
to learn how to use them you know. We want something simple. (Supervisor 1, #349)
Aaah, yes they are often completed incorrectly or inaccurately aah and I think it’s
just, it’s a result of haste how quickly I can do it, how little time I have to actually
read a question and do it and old habits die slowly. (Manager 1, #384)
Proper design of data collection instruments through participatory methods serves to ensure
that data gathered is well defined and accurate (Kusek & Rist, 2004). The quality of data
collected by the agency could be improved if an open communication was to be created
between the DSD and the agency with the view to obtain participation of those involved in
the data collection process. Moreover, staff training on the use of those tools and templates
could enhance the quality of data collected.
5.8.6 Soweto Care System as the Agency’s M&E System
As was alluded to in Phase 1, the agency has adopted the SCS as its computer-based M&E
system. This system creates ease in the compilation of process notes and storage and retrieval
of clients’ demographic data:
Ooh I also forget to mention that we have what we call a Soweto Care System, which
is somehow an electronic based planning tool which is similar to your diaries and
everything but it is very comprehensive because it captures absolutely everything
125
about the client, about the organisation about the frequency of contact about the next
court date about the lapsing of the order, about the significant others, you know. So,
even though it is somehow boring to use it, because it is electronic based and social
workers are not IT personnel you know, but I think these tools are very useful indeed
and sufficient in guiding service delivery. (Supervisor 1, #328)
The SCS also generates various reports which are useful in informing the agency about
demography of clients, caseloads, services rendered and so forth. Nonetheless, the SCS could
still be improved so that it is aligned with its intended purpose of generating M&E
information. It seems the SCS is in operation without guidelines to regulate how it should be
used as an M&E tool. As such, participants feel that the SCS can be manipulated to measure
what it is not intended to do (such as employee performance). Moreover, it is regarded as a
complex system which has taken social workers a long time to familiarise with. Its effective
use has also been hampered by shortage of computers implying that information on the SCS
is not always up-to-date. The fact that the SCS does not have a qualitative aspect in terms of
reports generated presents another shortcoming. Participants feel that the SCS could also
have been designed for the child welfare sector in the first place since it is now becoming a
time consuming process to adapt the SCS to the needs of the agency and the child welfare
field in general:
The system is too complicated and I think it somehow gets abused by the management
which is why social workers and supervisors hate the system. It is like a control tool.
It is like a camera on the social worker to say what are you doing? (Supervisor 1,
#684)
We have the Soweto Care System I think they just have to improve that computerised
system so that it becomes more efficient and is able to generate statistics on its own
rather than doing statistics manually... I’m sure that computer statistics will be more
accurate stats than the ones that would have been done manually. (Social worker 1,
#410)
It [SCS] has got so many other useless things that are not needed. That system should
have been informed by the objectives and goals of the organisation. (Supervisor 1,
#698)
It is not tailor-made to meet the needs of the organisation. It’s just a system which
was designed I don’t now for who, what sector. (Supervisor 1, #719)
126
We have the Soweto care system which I would also say would be focusing heavily on
numbers. I mean you can see how many home visits your social workers are
conducting but it wouldn’t tell you anything about the quality [of services rendered].
(Manager 1, #331)
It [SCS] doesn’t talk much to the effectiveness of the services in terms of quality... it
gives you [information which] is taken from the forms that are already in existence so
it’s just quantity more than the narrative. Yah, so it serves the same purpose as the
hard copy forms. (Manager 2, #397)
The agency could make effective use of the SCS if data generated by the system is analysed
and there is equal commitment by different units to ensure that information of the SCS is
correctly captured and kept up-to-date. The SCS could also capture qualitative information
which would be useful in informing the agency about its performance and point to areas of
improvement.
5.9 M&E Data Processing, Analysis and Use
Findings presented in the documentary analysis phase clarified how data is collected by
social workers and how it is handled up the chain to supervisors until it reaches the
management level. However, questions were raised on how M&E data is being utilised in the
agency to improve programme performance, and review and adjust agency strategy. Data
analysis enables the agency and its stakeholders to keep track of what is being done and what
is being accomplished (Lewis, Lewis, Packard, & Souflêe, 2001). It was brought to light
through the interviews that, in times past, the agency used to analyse ‘statistics’ for
establishment of trends, sourcing for funding, lobbying and advocacy purposes:
I think especially for the organisation our advocacy department was the one that was
responsible for collecting that and it’s because our advocacy department works with
other organisations and other stakeholders and other government officials, in terms of
lobbying for funding for children or all sorts of services that would benefit the
children so they would analyse that. We would just collect the factual data in terms of
the figures and then they do the analysis in terms of the trends as well. By looking at
the trends, it sort of shapes the services that we may need to venture into in order to
stay current and relevant to what’s happening out there. (Manager 2, #477)
127
Having further explored this matter, it was clarified that the agency no longer does any
analysis of the data that is collected on a monthly basis:
I don’t think anything specifically. If you look at the whole agency, they are collated
all the statistics are collated by somebody in accounts but you can only read them as
they are and say ok this is how many cases in this category... So we do a compilation
here for foster care so you can isolate foster care and have a look at it from an
isolated perspective but umm no there is no sort of somebody sitting doing an
evaluation. (Manager 1, #446)
Look, I’m not sure whether somebody higher up there analyse those stats but if they
do we certainly don’t get the results of that analysis to say are we achieving the goals.
To me it’s just compilation of stats. (Supervisor 1, #426)
Ummm the statistics, I don’t honestly understand the relevance of them because I
don’t know where they go from once they leave this department and who is looking at
them and who is analysing them and who is making any sense of them because we
never get any feedback umm from anyone... They just sort of disappear into
cyberspace and you don’t actually know what you are doing with them. (Supervisor
2, #191)
The above quotations confirm that there is no analysis of the M&E information that is
collected on a monthly basis. The following sections will consider the role of supervisors and
managers in the M&E system of the agency.
5.9.1 Role of Supervisors in the M&E Process
Supervisors have a major role to play as they determine the quality of services that are
rendered by social workers (Kadushin & Harkness, 2002). Since the agency has no tools to
measure the quality of services being rendered and does not necessarily track whether
services are being rendered from a developmental perspective, supervision of social workers
is the agency’s main mechanism to cover this gap. Supervisors promote competent social
work practice and they thoroughly check the quality of reports, ‘statistics’ and services
rendered by social workers. This finding is consistent with the importance of supervision in
social work as described by Kadushin and Harkness (2002):
128
You see social workers get supervision and that’s part of monitoring from the
supervisors’ point of view. (Manager 2, #261)
Nevertheless, we also strive to render quality services... our reports are canalized by
our supervisors and our managers. You don’t just compile a report and send it to the
court. It is canalized and I think it is an intense canalization. (Social worker 1, #121)
Inasmuch as social workers undergo intense supervision, the agency shortcoming is that
processes at this level are not recorded and there are no reports produced except the
supervision notes attached per individual case. As such, it is difficult to measure the quality
of services since there are no reports produced that could be analysed.
5.9.2 Role of Managers in the M&E Process
As highlighted earlier, it is only managers who are involved in M&E planning. They are
responsible for compiling the business plan and handling M&E information collected by the
social workers and submitted to them through supervisors. Managers are also involved in
checking for quality as they also canalise reports and statistics submitted to them by
supervisors. They also engage in file audits and attend panels to monitor whether services are
being rendered from a developmental perspective:
And as a manager, you do audits of the files because we work with files and whatever
the social workers do is documented on files so that can be used as monitoring tools
as well. And we also have panels with some of our stakeholders especially the people
that we place children with. And that also helps us in terms of monitoring. (Manager
2, #263)
I check the statistics that come in and half the time I have to say to people this and
this and that and then they go back and fix it. And the fact that they have fixed it, it
tells me that if I was to pass it through, then it wouldn’t be effective or adequate or
efficient. (Manager 2, #337)
Since managers do not have M&E expertise, it is difficult for them to effectively perform this
role. This suggests to the student that M&E is not fully integrated into the core business of
the agency. As such, focus is placed on managing procedures and performance on social
129
workers rather than collecting M&E information and using results to improve service
delivery:
The emphasis is more on processes than monitoring and evaluation because we are
looking at what is the flow of work in the organisation but the impact of that flow,
how things would flow is not necessarily measured. Yah so I don’t think there is
emphasis on M&E. (Manager 2, #314)
The agency generates voluminous complex quantitative data which requires scientific data
analysis tools to make sense of. Since the agency does not have these tools, there is hardly
any data analysis being conducted. Consequently, the agency also makes minimal use of the
data collected. The following section will explore the M&E reports generated by the agency
and outline how M&E information is reported by the agency through specific structures such
as the six-monthly progress reports and the annual report.
5.10 M&E Reports
According to Rabie and Cloete (2011), reporting demonstrates accountability and delivery on
promises. It is against this premise that the agency reports to the DSD after every six months
concerning the progress it has made towards achieving the objectives as stated in the yearly
business plans. Kusek and Rist (2004) also contend that reporting creates institutional
memory, provides organisational learning, informs decisions, and promotes understanding
and support from stakeholders. The agency expresses its obligation to report to its various
stakeholders (such as funders, partners, employees and clients) through the annual report.
Moreover, the agency utilises the newly adopted SCS to generate reports which could be of
importance to the improvement of service delivery and the improvement of the M&E system
as a whole.
The six-monthly progress report provides a platform for the agency to report on the progress
made over the year. It helps the agency put thoughts into the effect of the services they are
rendering (Rabie & Cloete, 2011). Moreover, it allows them to rationally look at the
challenges they are facing in delivering community-based services and explore possible
solutions (Kusek & Rist, 2004). Nonetheless, the interviews revealed that the six-monthly
progress reports mainly contain more quantitative information which really does not speak to
130
the quality of services being rendered by the agency or the effect of the services in terms of
outcomes and impacts. Since the six-monthly progress report is a reporting structure
prescribed by the DSD, participants feels that this structure is limiting and does not allow the
agency to be creative in terms of reporting on qualitative information. Participants would
appreciate a situation where the DSD would respond to its challenges and limitations. The
student also found out that the six-monthly progress reports are considered confidential. As
such, practitioners who are responsible for implementing the plans and collecting M&E data
feel excluded as they are hardly made aware of the effect of their services and whether, in
fact, they are achieving the overall goal and objectives:
One wonders when you compile a progress report and they ask you things like
challenges and plans of action, aah who takes notice of it what your challenges are
because in the next six months on a progress report you are saying the same thing you
know. (Manager 1, #502)
The progress report is nicer to do. I like to do the progress report because that’s
where you get to talk about your successes and your challenges that you would have
met in the year and it actually makes you think about what you can do differently the
following year. Although the challenge or the frustration may be that as you write it,
you hope that the government department is going to take note of the challenges...
Much of the challenges are in there year in year out. Nothing comes out of them.
(Manager 2, #568)
The manager’s report? I’m not sure if I’m privy or if we get to see the report so on
that one I wouldn’t say ukhuti [that] there is adequate because it’s something that I
will never see... I think the manager’s report which I think I noted above should be
made available to us... so that we know what is contained, what were the goals, what
were the objectives, and for us to know if we managed to meet those goals and
objectives. (Social worker 4, #448)
Managers’ reports? I have never seen one in my entire life. Besides being told that I
am compiling my manager’s report and besides being told like in the manager’s
report there is this and that. What I know is that we provide information upwards to
the manager so that she can compile the manager’s report but that information is just
numbers, which to me I do not know what those numbers are. (Supervisor 1, #640)
131
Given this background, participants feel that there are limitations to the usefulness of the
reports generated by the agency especially that they capture, mainly, quantitative information.
Participants feel that the agency could benefit more if the information contained in the six-
monthly progress reports could be both qualitative and quantitative, and if it could be shared
between management and the practitioners responsible for implementing the plans:
I would love to see the organisation producing maybe especially those six-monthly
progress reports and give every social worker a copy so that we see where we are
going wrong, where we what we are achieving and what we are not achieving. (Social
worker 1, #389)
Concerning the annual reports, the participants felt that this reporting structure does not really
add value to the agency since it is a summary of the information contained in the six-monthly
progress reports:
I mean the annual report talks to the highlights of what has happened in the team. The
figures, yes, would be there because you would have gotten them from the six-monthly
progress reports. And they are what they are. So if they are wrong in the progress
report, they would be wrong in the annual report... And also, I wouldn’t say it’s
qualitative. It’s just how we brag about things. I would say the annual report it
doesn’t add anything to anything. (Manager 2, #807)
However, other participants indicated that the information they provide for the annual report
is different from the six-monthly progress reports. They did not understand how the agency
comes up with information contained in the annual reports. As such, participants feel that the
information contained in the annual report could be misleading:
Umm because every time it comes to annual reports, you are actually asked for
information specifically for the annual report. They don’t seem to be gathering that
information from the statistics that we have provided throughout the year... You give
those stats specifically for the annual report. (Supervisor 2, #429)
As far as the SCS is concerned, participants felt that this system has got potential of
producing information which could be valuable to the agency. However, at this stage there is
132
a lot of scepticism associated with the SCS especially that it was recently adopted by the
agency and is in the process of being tailored to meet the needs of the agency:
You see the system I feel it is a stagnated system which is still at its infancy and has
got a stunted growth which will never grow from there. It is where it is and you know
we try by all means to improve it by providing extra information, the next thing it’s, it
has got so many mistakes and stuff like that such that the reports that are produced by
that system we can’t rely on them. (Supervisor 1, #666)
In light of the presentation made of the findings obtained in this phase, the student explored
the training needs that the agency needs as well as the recommendations to improve the M&E
system for child welfare field. The following sections will therefore highlight the training
needs as well as the recommendations generated in order to improve capacity for M&E in the
child welfare field.
5.11 Training Needs
Since staff provide the vehicle through which the purposes of the organisation are carried out,
different activities and services require that these members of staff be competent and talented
in various ways (World Bank, 2006). As highlighted in the documentary analysis phase, the
agency offers training sessions whose focus is to promote continuous professional
development and to develop skills among practitioners. Nevertheless, the participants felt that
the agency could build on that by exposing staff to M&E training, analysis of M&E data
generated by the agency and training staff across the agency on tools and templates used to
generate M&E data. This could go a long way in enhancing data integrity and promoting the
collection of data that could be used to improve programme performance and service
delivery. Participants also felt that training could be offered on a continuous basis due to high
staff turnover experienced in the child welfare field (Lawrence et al., 2013):
I think the organisation needs to expose us to trainings regarding M&E because we
haven’t yet received training of this sort so I think it is good that the organisation
exposes their social workers to this kind of training because we are not really well-
versed in what really is monitoring and what really is the evaluation. (Social worker
1, #250)
133
I think the first which I think is very core is training not only in just writing these
figures but training in analysing this data. After gathering the data, eeh we also need
to know how to analyse this data so that we can know if our services [are] effective.
Are we meeting our intended goals? Are we failing? (Social worker 4,#356)
I said even the statistical forms umm you know I think there is a very different way of
doing those forms. Everyone is filing them just according to how they are seeing them
umm and not really according to the standard umm way of looking at it. I think one
does need to have regular training and particularly we get new staff all the time.
(Supervisor 2, #309)
5.12 Recommendations for Improving M&E in the Child Welfare Field
Other than the recommendations generated and proposed for each specific theme, the student
also elicited information from the participants on how the M&E system could be improved
for the child welfare sector in general. The participants suggested that, foremost, child
welfare should strive to develop better M&E policies and guidelines. This should be followed
by clear M&E structures within agencies. M&E systems in child welfare need to be designed
specifically to meet the needs of the sector. This implies that tools should be standard,
scientifically developed and should generate information that is both qualitative and
quantitative using an electronic platform. Participants suggested a unified system between the
main funder (DSD) and the agencies providing child protection services. Moreover,
participants felt that service users are often left out in M&E processes, whereas they should
be included. As such, M&E should focus on the beneficiaries and should be consultative,
participatory (World Bank, 2004) and utilization-focused (Patton, 2004). Furthermore,
internal processes should be efficient so that there is accountability and collaboration between
units. Child welfare agencies could strive to analyse data currently being collected and to
produce more reports from which they could draw lessons and improve service provision and
the management of programmes (Kusek & Rist, 2004). The following quotations are
provided in support of these recommendations:
It will be a tiresome process for them to speak to our clients and go to each and every
client and ask [them about the] the quality of services that are rendered to them.
Although it can be, it can give a broader picture of the services that we are rendering
134
to them but I think it’s quite tiresome and time consuming to do that. (Social worker
2, #192)
I think the only report that should be important is the monthly report instead of
having quarterly reports at least they will be able to make you set a different pace for
what you do. (Social worker 3, #289)
Again I don’t know what else they [the agency] can collect because I don’t know how
much they have got at the moment, how much they are gaining from what we [are]
giving them at the moment. (Supervisor 2, #284)
I think if there was to be clear monitoring and evaluation structures and tools that are
universal in the organisation; [if we had a] general framework that we all work from,
that would work because I would say my department is doing well. But it’s subjective
because I’m the one doing it and I’m maybe coming up with my own kind of
evaluation and monitoring tools and yet if everybody is using the same thing that
might just work better. (Manager 2, #302)
5.13 Conclusion
Through semi-structured interviews, this chapter has considered the agency’s M&E system as
it works in practice. It has verified some of the findings generated in the documentary
analysis phase and answered questions which arose from the first phase of data collection.
The agency has various ‘high’ level documents which, to some extent, provide overall
direction on how community-based services for OVCs should be monitored and evaluated.
The agency has specialised units, structures and personnel responsible for implementing
OVC programmes and collecting relevant M&E data useful for reporting purposes to the
main funder (DSD). In spite of the challenges generally experienced in the child welfare
field, the agency is resourceful and has managed to be in existence for more than a century.
Nevertheless, findings in this phase of data collection (semi-structured interviews), coupled
with information gathered from the documentary analysis phase reveal that the agency has a
structure that places excessive emphasis on service delivery, organisational processes and
protocols rather than the role of M&E in informing service delivery improvement. This
situation is compounded by the fact that the agency has no dedicated personnel who focus
and give expert attention to M&E. There is lack of consultation of staff regarding M&E. A
135
lack of clear M&E policies and guidelines entails that the agency is operating without
articulated, planned and purposeful guidelines. Resultantly, practitioners at the point of
service delivery cannot effectively execute their M&E duties especially because they lack
training in M&E and that they render services while being unaware of the overall goal and
objectives of the agency’s M&E system. M&E tools used by the agency do not generate
adequate and reliable M&E information and there is no analysis of information collected. In
fact, the relevance of tools was questioned by the participants and the fact that there are
higher expectations of service delivery and M&E data collection when the agency does not
have adequate resources (such as computers). By and large, there is resistance to M&E as it is
perceived as a mechanism of social control and performance evaluation by staff.
Consequently, data gathered is insufficiently used. These foregoings highlight the limitations
and inadequacies of the agency’s M&E system.
The following chapter shall illuminate on the findings emanating from the focus-group
discussion, which was conducted to obtain perceptions of participants regarding the findings
generated through the documentary analysis and the semi-structured interviews.
136
Chapter Six: Phase Three Findings: Participants’ Overall
Perspectives and Recommendations
6.1 Introduction
The preceding two chapters presented the findings that were drawn from the documentary
analysis and the semi-structured interviews, respectively. The current M&E system of the
agency was documented, including the way the system works in practice. The purpose of the
focus group discussion was to obtain the participants’ perceptions of the overall findings; to
validate the findings; and to augment them with their own interpretations. In addition, the
student intended to explore options for improving the agency’s M&E system. While the
previous chapter reported on individual interviews conducted with social workers,
supervisors and managers, the focus group discussion provided participants with the
opportunity to hear the perspectives of others, to engage critically with the findings, thereby
increasing the trustworthiness of the study. Burns and Grove (2009) concur with this
approach to increasing trustworthiness.
As described earlier in Chapter 3, Section 3.6.1, the participants for the focus group
discussion were purposively selected and comprised two managers, two supervisors and four
social workers initially interviewed by the student. Their biographical details are presented in
Tables 5.1 and 5.2 in the previous chapter. The student also incorporated the agency’s
advocacy coordinator who uses agency M&E data (statistics) for lobbying purposes and
influencing child welfare policies. The advocacy coordinator is a white female who has been
in the employ of the agency for 37 years. Thus, she has vast experience in the agency and the
child welfare field at large.
The focus group discussion was structured in such a way that findings were presented to the
participants and discussed based on the following themes:
Overall M&E policy and guidelines
Organisational structure
Knowledge of the principles governing community-based services for OVCs
Community-based services
137
M&E tools and processes within the agency
M&E data processing, analysis and use
Reporting
Training
The following is therefore a presentation of the findings emanating from the focus group
discussion. The format allowed for a smooth flow of the discussion, which subsequently led
to the generation of recommendations.
6.2 Overall M&E Policy and Guidelines
The point of departure in this study is that policy is important in granting child welfare
organisations the go-ahead to make decisions that guide the course of action to be followed
by child welfare practitioners (Rabie, 2011). Having explored whether the agency has an
overall M&E policy and guidelines, all participants agreed with the finding that the agency
has no overall policy, guidelines and indicators to govern how community-based services
should be monitored and evaluated. The participants in the focus group agreed with the
finding that the DSD (as the main funder of the agency’s OVC programmes) has a prominent
role to play in guiding decision making, setting rules for service provision, identifying the
overall procedures, outputs and outcomes to be achieved including documentation and
reporting requirements such as business plans, submission of monthly statistics all of which
govern the overall M&E system of the agency. Realizing that such is the reality of how the
agency’s M&E system works in practice, one participant concluded that “It’s a difficult thing
[to formulate and operationalise [our] own policies and guidelines] because we [the agency]
need the funding to operate and the people giving the funding pull the strings [controls the
agency].”
Moreover, participants acknowledged that the agency’s M&E plans are not articulated well
for effective OVC programme implementation, monitoring and evaluation. The conclusion
that was drawn in this regard is that there “is a new consciousness emerging [about the
importance of M&E in the agency] but it might... not have gone as far as having a proper
policy.”
138
The focus group discussion also helped confirm the student’s impression in Chapter 4,
Section 4.1.2 that the agency is aware of the shortcomings of its M&E system, hence steps
are being taken towards the development of the M&E system:
I think it [M&E training] was a reflection of a concern about the need in the agency
for such a system and yah so we haven’t carried it through. (Participant 1, #129)
We had the workshops for managers on monitoring and evaluation not too long ago
ummm and ummm when [the assistant advocacy coordinator] was here and was
intending to stay… she did monitoring and evaluation work and she had been
involved in it quite intensively and it was intended that she should continue with that.
(Participant 1, #86)
From this discussion, it can be deduced that the agency has taken steps towards the
development of its M&E system, but this has not yielded much results due to a lack of follow
through on the agency’s commitments. A loss of expertise and staff turnover resulted in the
M&E processes not being implemented.
Participants agreed with the finding that amidst the agency’s lack of a clear M&E policy and
guidelines, the annual business plan is instrumental in providing guidance on what objectives
to achieve and targets to meet (planning and monitoring). Nonetheless, the annual business
plan is rigid in structure, leaving little room for the agency to make programme initiatives and
capture other forms of information that could be valuable to the agency.
Having explored the above findings with the participants, the student proceeded to present
findings related to the organisational structure.
6.3 Organisational Structure
An exploration of the manner in which the agency is organised and how it affects service
delivery was discussed. All participants concurred with the finding that the agency has a
bureaucratic structure, which utilises a top-down approach when giving instructions and a
bottom-up approach for staff to implement the plans and collect data. Participants agreed with
the finding that the M&E function is embedded in the organisational structures and functions.
There is a lack of M&E expertise and a specialised M&E unit that is responsible for this
139
function. Even though the majority of the participants agreed that there is poor emphasis on
M&E by the agency, it was clarified that the agency is conscious of the need for M&E
improvement in the agency.
It was agreed that embedding the M&E function within the agency structures is insufficient
and does not effectively mainstream M&E in the operations of the agency. One of the
participants identified the need for buy-in of staff and supervisors to ensure that M&E is
effectively mainstreamed throughout the organisation. This is what Participant 2 said:
I agree with you. My emphasis is going to be on buy-in from staff and supervisors; it
can only come from staff and supervisors not the top-down approach, which we
always tend to resort to, which I say “you must” but you say “I won’t, I can’t”. So,
for me, to have buy-in your staff have to be involved on that level. (Participant 2,
#1113)
There was appreciation for the fact that the agency requires technical M&E expertise.
Participant 1 identified the knowledge gap in this area:
As I said we [the agency] were wishing that [the assistant advocacy coordinator]
would take it forward but then [she] left us. And now there is another person who has
taken [her] place and there have been questions as to whether she will get into
monitoring and evaluation. (Participant 1, #138)
A majority of the participants were in agreement with the finding that the agency structure is
not arranged in such a manner that units can coordinate their activities seamlessly and assist
each other in the achievement of objectives. It was however clarified that different specialised
units within the agency do cooperate with each other as there is liaison between units. Data
management is often compromised when there is poor coordination between various
specialised units:
... [file and service transfer] depends on how the file is presented so often files will go
back, and go back, and go back and go back… So the attention to detail and the
quality of the services are often rather lacking. (Participant 2, #225)
140
This quotation also accentuates the finding that the agency places excessive emphasis on
administrative processes rather than M&E per se. Emphasis placed on the administrative
functions (such as compilation of process notes) overshadows M&E functions. Whilst
administrative duties are an important aspect of service delivery, more attention could be paid
to the extent to which services are making a difference to beneficiaries.
Having clarified on this aspect, the discussion proceeded by exploring the findings which
follow below.
6.4 Knowledge of the Principles Governing Community-Based Services
for OVCs
The World Bank (2006) postulates that staff provide the means through which the purposes
of the organization are carried out. Different activities and services require that these
members of staff be competent and talented in various ways. For this reason, this study also
elicited information about the participants’ understanding of the terms ‘M&E’, ‘indicators’
and the ‘developmental approach’. Additionally, the study intended to explore the
community-based services being rendered by the agency. Findings related to this were
presented in the focus group discussion as well.
The participants were unanimous about the finding that their understanding of M&E is
skewed towards funding and accountability, hence M&E is not necessarily about programme
improvement in the agency. There was no dispute about the finding that there is a general
lack of understanding of what indicators are. In fact, during the focus group discussion,
participants made remarks which substantiated this finding and highlighted the need for M&E
training in the agency:
Are those not indicators? Are we not confusing indicators for outcomes? I think that’s
what you are trying to say. (Participant 3, #856)
But I think there are pockets of information in the agency which are indicators of
success in the work we have done... and some of the outputs are also outcomes so you
could take the number of children placed in adoption as an output but it’s also a huge
outcome and it reflects an enormous amount of work. (Participant 1, #874)
141
The question that was asked by one participant: “Are you saying that we lacked an
understanding of the developmental approach? You found that supervisors had a fair
understanding but we didn’t?” suggests that managers perhaps lack an awareness of their
shortfalls. It could also imply that managers expect themselves to have more information than
the rest of the staff, which is in contrast to the findings of this study. The student’s
explanation gave the participants who were confounded by this finding a chance to introspect.
Subsequently, participants agreed with the finding that there is an inadequate knowledge of
the developmental approach among participants as the majority could not operationalise it
within child welfare.
Participants concurred with the finding that in light of the knowledge gap caused by the
introduction of new policy and approaches to child welfare, practitioners (social workers and
managers alike) need to acquaint themselves with the principles of developmental child
welfare practice. The agency has a good platform for educating and providing guidance to
practitioners (i.e. monthly workshops, information sharing sessions and training sessions),
and this platform could be optimally used to bridge the knowledge gap.
Having settled findings related to the knowledge base as discussed in this section, the focus
group then moved on to discuss findings related to community-based services.
6.5 Community-Based Services
Community-based services give effect to OVC programmes and the quality of these services
determine the extent to which goals and objectives for OVCs are achieved (World Bank,
2006). On the whole, participants agreed that the agency has a wide array of community-
based services. In light of the funding received by the agency, the resources and the budgets
allocated for OVC programmes, the agency is able to appropriately respond to the OVC
needs. The agency is, however, constrained by resources (which are financial, material and
infrastructural). For this reason, the agency finds it challenging to expand its OVC
programmes and to implement new initiatives.
Nevertheless, the finding that there are no specific programmes that offer home and
community-based care or that deal with issues of poverty reduction, economic development
142
as well as issues of support and empowerment, was contested. A majority of the participants
were convinced that the agency has holistic and core OVC programmes that address issues of
poverty, and aim to provide support and empowerment through the agency’s community-
based centres:
I’m just thinking of the services [rendered by a specific community-based centre
belonging to the agency], which do include some poverty reduction measures and I
think they have a holistic approach. (Participant 1, #278)
And [a specific community-based centre belonging to the agency] that’s its only
function in poverty reduction with an OVC focus as well. If we [are] training people
in the city then they are able to take more responsibility for the children as well so
that falls in. (Participant 3, #306)
The agency clearly has home- and community-based care programmes whose aim is to
address poverty, foster economic development and render supportive and empowerment
services. Such programmes are offered through community-based centres situated in various
areas of Johannesburg (Soweto, Eldorado Park and the Central Business District) for certain
clientele. Nevertheless, the student realised that the services that are provided through these
centres do not necessarily reach children and families who are in the foster care system as
accounted for by the agency. Clients who benefit from these programmes do not necessarily
form part of the agency’s caseload. It is conclusive, therefore, that the agency does not have
comprehensive community empowerment programmes and a holistic approach to poverty
reduction. There is no evidence in the agency of M&E activities related to the poverty
reduction services offered through the agency’s community-based centres
Having clarified the aspect of community-based services, findings on M&E tools and
processes were presented and discussed. The following subsection presents what came out of
that discussion.
6.6 M&E Tools and Processes in the Agency
The Soweto Care System (SCS) is generally recognized as the agency tool for M&E. It is
expected that the SCS should assist the agency in M&E data capturing and processing.
143
Nevertheless, all participants acknowledged that there is unequal commitment to the SCS by
different specialized units:
I tend to agree we underutilize this Soweto Care System. Umm firstly because we are
a smaller department but I think if I look at [our community-based centre] we are
really lagging behind. Really, really lagging behind so I don’t think it’s [an]
inaccurate reflection. So I would say we need to update it [SCS]; other departments
are being much more on the ball and are using it correctly. So there is truth in that
definitely. (Participant 3, #490)
It was further agreed that the SCS was not designed as an M&E system for child welfare. As
such, the process of tailoring the SCS is time-consuming. The agency required, from the
onset, an M&E tool designed specifically to meet the needs in the child welfare sector.
Nonetheless, the participants were of the same opinion that the SCS has potential for
effective M&E use in the agency:
I think you are correct. It [SCS] wasn’t designed for child welfare. However, its
development has become something very useful and inputting the data is time
consuming but in the end – I think we look at all the stats, we talk about all the stats
and the time it takes – but in the end if it’s done properly it can save us a lot of time,
which we can then use for, in my view, quality [service delivery] instead of facts and
figures. (Participant 2, #503)
It is, however, noteworthy to mention that participants lacked confidence and could not
substantiate how the SCS could be used as a tool for programme improvement in the agency.
Participants concurred with the finding that the SCS seems to be replacing manual processes
that have been traditionally used by the agency in the generation of data required for internal
purposes and for submission to the DSD. Essentially, it is not clear how the SCS is to be used
as an M&E tool. For example, one participant reiterated the problems associated with the
output driven approach: “In the current form of the Soweto Care System, what we only
[produce] are numbers or figures... but for the organisation to determine the effectiveness or
the quality of the work you are doing, it fails on that one. (Participant 4, #603)”. Another
participant expressed scepticism as to whether the SCS would ever be able to measure the
144
quality of services rendered: “Will the Soweto Care System ever be able to get that [measure
the quality of services rendered]? Probably not. (Participant 3, #613)”.
Participants did not adequately show an appreciation of the importance of procedure manuals
on how information from the SCS is to be used to inform programme improvement. Rather,
participants focused more on the lack of instructions on how new users might use the system.
Participants highlighted the need for more resources (e.g. computers), staff buy-in, and
analytical capability to use the information obtained from the SCS to improve service
delivery and to enhance the agency’s M&E system:
I think it’s never gonna be an effective tool unless if everyone gets their computer. It’s
frustrating for social workers because they can’t get to do their Soweto Care System
when they need to do it so they still complement it with the manual [way of writing]
process notes. (Participant 4, #537)
You know I think there isn’t, there isn’t a lot of buy-in from a lot of supervisors and
staff (Participant 2, #565)
It’s like what you said in your findings, we don’t use it correctly. We get the stats but
we don’t do much with them (Participant 3, #576)
All participants agreed with the finding that there is hardly any participation of the staff (i.e.
social workers and supervisors) in M&E planning since it is perceived to be a management
responsibility. Rather, staff play a major role in the collection of data; a role which they
regard as peripheral and often conduct with resentment:
You know there isn’t buy in by the staff because they don’t understand the relevance
of it, where it’s going and who is gonna interpret it and what’s it [is] gonna be used
for. So that’s the difficulty they just do it in a negative way. (Participant 4, #411)
Participants highlighted that M&E data collection tools are not standardised as they are
always changing in accordance with the demands of the funders.
The Department does the same, the stats are like this, next time we are changing to
this. So it makes no difference what you have collected because there is no sort of
standard format that a person says “all donors will be given this”. It’s what the
145
donor wants. And we tailor it to suit that because we need that money, you know.
(Participant 2, #734)
The unanimity of participants as regards the lack of proficiency in the manner in which DSD
conducts itself was striking during the focus group discussion. Participants stressed that there
is not enough communication between the agency and the DSD, which often leads to poor
understanding of why certain data required by the DSD is important:
There is not enough communication, perhaps, about how it all hangs together. And
not enough coming up from the ground in terms of what is needed right there where
the surface is, as part of all of this. (Participant 1, #437)
I don’t think it’s irrelevant. I think that there is often not so clear [an] understanding
of why it’s relevant over here and then how it connects up. (Participant 1, #427)
Poor communication between the agency and the DSD was cited as the reason why the
design of tools for M&E data collection by the DSD (for child welfare agencies) was not
participatory. Due to poor communication, tools are independently designed by the DSD and
imposed on child welfare agencies. Moreover, participants felt that the DSD lacks the
dexterity required in designing M&E tools:
Because a lot of them [tools] are designed by DSD, so that’s where the problem is.
They are not the brightest bunch when it comes to M&E... They often miss the goal.
(Participant 2, #365)
I think they [DSD] think we work in the same way or it’s just eeh I don’t think their
tools are very eeeh precise (Participant 3, #371)
Resultantly, the submission of statistical data was seen by participants as DSD’s “own
persistence of accountability” not necessarily the means to generate data useful for M&E
practice.
Other than the DSD’s lack of proficiency, the focus group participants also expressed their
concern over top management of the agency being overly involved in the specialized units,
which disrupts their autonomous functioning. Participants emphasized the difficulty that lies
in conducting M&E duties and coming up with strategies for improvement given that units
are burdened with demands for other things which they feel are irrelevant to them:
146
And I think from inside, I agree with that, but internally while we do function as you
said in units, with our own objectives, internally, everybody interferes with anybody
else’s functioning in deciding what you need or your staff, or your supervisors... there
is no time to do what you think should be your own internal training, your own
internal involvement of your own staff because the moment you decide to do that
somebody will come from outside your department within the agency and say, “Your
staff needs to do this and your staff need to do that.” (Participant 2, #381)
Findings on this aspect of M&E tools and processes within the agency sparked a debate
which subsequently led to the conclusion that there are both internal and external factors that
explain the poor design of M&E tools and the organisational complexities that have a bearing
on the workings of M&E systems within the agency. Closely linked to this aspect of M&E
tools, findings on M&E data processing, analysis and use were presented and discussed as
discussed below.
6.7 M&E Data Processing, Analysis and Use
The discussion on M&E data processing validated the finding of this study that there is
overemphasis and fixation on outputs by the agency. By and large, participants seemed to
downplay the importance of quality measurements in child welfare services. Rather, the
importance of quantitative information (output indicators) was accentuated because indicators
of quality are presumed to be difficult to measure.
I just wanted to reiterate what they have raised that as a welfare organisation in child
welfare services we ought to rely more on quantity because of quality [of service
delivery] we will never be sure. We won’t be able to measure quality. So I think that
quality we can just assume from the quantity that we are providing on the services.
(Participant 9, #631)
This finding is a reflection of the deficiency of tools in the child welfare system to measure
qualitative aspects, hence child welfare agencies simply resort to measurement of outputs. In
spite of this, participants agreed that the collection and analysis of both quantitative and
qualitative information is more consistent with the child welfare field since that would
capture the quantifiable and intangible aspects of child welfare services. Moreover, the
147
collection of quantitative and qualitative data would enable the agency to evaluate progress
being made in implementing OVCs policies and approaches to child welfare.
In Chapter 5, Section 5.8.6, it was highlighted that participants felt that the SCS could be
manipulated to measure what it is not intended to (such as employee performance). During
the focus group discussion, it became clear that monitoring tends to be confused with scrutiny
or performance appraisal by management. Whereas management should be concerned about
the collection and analysis of information on how a particular OVC programme is
progressing and its effect, the focus seems to be on how social workers are delivering
services:
You know I can monitor [through SCS] how many times you see a child but I can’t
monitor what you say to the child. And I can only, and a supervisor and another
social worker can only assist in the quality of the services by then looking at it
individually. (Participant 2, #589)
M&E processes in the agency could be improved if managers were focused more on goal
achievement, leaving supervisors to direct how services should be implemented.
The finding that there is no data analysis in the agency was generally agreed on by most
participants. The majority of the participants were of the same mind with the student that
M&E data generated by the agency is underutilized and does not necessarily resonate with
service delivery improvement:
I think you know what, honestly for me… it’s like what you said in your findings, we
don’t use it correctly. We get the stats but we don’t do much with them. I think there
has to be proper training we need time to actually sit with these things and implement.
And we just don’t have that time. (Participant 3, #577)
I was just going to say what always interests me is that we continue to do our stats
and giving these figures but suddenly when the agency needs figures for, let’s say, the
business plan for example then they come back to you then and say “can you provide
these figures for this time period.” So what happens to all the figures we have been
giving all along, every month? (Participant 4, #720)
But that’s what I am saying, if we are giving the monthly stats forms where are they
going? So that’s someone sitting with that information. (Participant 4, #762)
148
Nonetheless, one participant contested this finding and highlighted that there is some form of
data analysis conducted by the agency, even though it may be unclear or inadequate:
Generally, I wouldn’t say that there is no data analysis. There is some data analysis
conducted for a specific purpose but it does not connect with service delivery or
service delivery improvement (Participant 1, #698)
Well, every year the agency in its annual report states what it’s been doing, how much
it’s been doing this type of work and that comes from the stats that’s been collected
over the year. So that is an analysis. And when everybody goes back and writes their
business plan to the Department they begin with some data analysis but connecting up
with service improvement that’s what’s not happening. (Participant 1, #705)
On the whole, participants were of the same mind that M&E data generated by the agency is
not being processed, analysed and used in such a manner as to inform service delivery
improvement. Even though it was argued that there was some form of data analysis and use,
the majority of the participants maintained that more could be done to show that data analysis
and use were evident in the agency. The following subsection, will consider the participants’
views of the findings on the agency’s reporting structures.
6.8 Reporting
The reporting structures used by the agency to report on progress made in light of the plans
enunciated in the annual business plan is a fairly straightforward process that works for the
agency. Participants went along with the finding that the two main reporting structures of the
agency (i.e. the annual report and the six-monthly progress reports) do not necessarily speak
much about M&E. Rather, these reports mainly relate to information on outputs:
There is supposed to be, I haven’t seen any of these reports in a long time now, the
annual report on your programme is supposed to have some content like output.
Ummm and then some important aspects would be how many children are adopted,
how many children are returned home… (Participant 1, #841)
Participants highlighted that other reports are sent to stakeholders as and when they are
required.
149
As highlighted from the onset of this study, the agency generates statistical data for internal
and external purposes. The external purpose is, essentially, reporting to the DSD on the
services rendered every month. However, participants unanimously agreed with the finding
that, once such data is submitted to the DSD, it is not understood how it is processed,
managed and used. All participants concurred that there is hardly any feedback received of
the reports and ‘statistics’ sent to the DSD and that this was not peculiar to the agency but
“the complaint from all of the NPO sector.”
Through the focus group discussion, it was clarified that the agency is able to measure and
report on certain objectives depending on how they are expressed (i.e. quantitative or
qualitative form) notwithstanding the poor feedback mechanisms by the DSD:
We mustn’t say we can’t measure the objectives… But it would be true to say we
cannot report on the achievement of the objective. If our objectives are phrased in
terms of quantity, that might be a deficiency in the objective (Participant 1, #911)
It was generally agreed by all participants that a gap exists, in that supervision and
administrative processes at supervisor level are not reported on, even though they
significantly contribute to the quality of service delivery and M&E data collection.
Participants also acknowledged the gap that exists, namely that no reports are produced for
internal purposes, that is, to give feedback to practitioners on progress being made pertaining
to set goals and objectives. M&E reports could be produced more frequently for internal
purposes, specifically, to measure the progress in implementing OVC policies and to inform
programme improvement.
The following section discusses participants’ perceptions of the findings related to training.
6.9 Training
Participants unequivocally agreed with the finding that there is a significant lack of M&E
skills among practitioners within the agency. In spite of this, the training platform continues
to be utilised for inculcating social work skills for professionals to deliver social work
services effectively. Emphasis on M&E therefore continues to be lacking in the agency.
150
To reiterate what was highlighted earlier in Section 6.2, the agency has a renewed awareness
about the need for M&E. This awareness has resulted in the agency taking initiatives towards
M&E training for supervisors and managers. Participants, however, agreed with the finding
that M&E training has unfortunately failed to equip staff with skills to effectively execute
their M&E duties:
I don’t think actually for me that it was monitoring and evaluation. It was, and it’s
purely my opinion, it was a one sided effort by her [the trainer] to say “none of you
know what you are doing!” (Participant 2, #109)
I think it was not really helpful at all... And also there were gaps. We had one session
and it was cancelled the next one was cancelled and no continuity. I mean it ended
abruptly so I really did not get anything from it. (Participant 3, #113)
A discussion of these findings related to training, suggests to the student that the agency
needs to invest in the training of staff on M&E issues. Moreover, it needs to identify
measures to ensure that M&E training is sustainable and equips personnel to effectively
execute their M&E duties. The following are recommendations that were made by
participants on how the M&E system can be improved.
6.10 Recommendations
During the focus group discussion, it was recommended that the agency focuses on the core
business rather than experimenting with other things for which it does not have the funding:
…if you don’t have the money and you must look at your core services as like being a
river. If you don’t have the luxury of other money or extra money, don’t dabble on the
banks just focus on your core services. And honestly for me I often say all we do is
dabble on the banks instead of looking at, and keeping the focus. (Participant 2,
#1093)
For the agency to effectively focus on the core business, that is service delivery and
instituting M&E measures, participants recommended that there should be single-mindedness
and staff should pull in the same direction:
151
I think we should just all be on the same page in the agency in terms of monitoring
and evaluation or really as I say we are fragmented. (Participant 4, #1099)
Collaboration of all parties at agency level depends on the agency’s ability to promote
participation and create buy-in. This can be achieved through, for example, the promotion of
a bottom-up approach in the agency structure where social workers and supervisors are
consulted and have a say in M&E planning. Participants also felt that it would be beneficial
for the agency to explain to staff why, when and where specific M&E data is required. The
resultant effect would be an emphasis and commitment to M&E in the agency:
Buy-in from staff and supervisors, can only come from staff and supervisors not the
top-down approach, which we always tend to resort to… So, for me, to have buy-in
your staff have to be involved on that level. (Participant 2, #1113)
It should be, and I know it’s a cliché but a buy-in by all the staff as to why we are
doing this and the importance of it because until we get that, nobody is gonna give
any great emphasis to doing it [M&E data collection] and see the purpose of it.
(Participant 4, #1100)
In response to the gaps identified in Section 6.8 above (lack of tools to measure quality of
service delivery and shortcomings in the phrasing of objectives), participants also
recommended that the agency comes up with measuring instruments for the established
programmes as a way of improving the M&E system:
We also need to ensure that in every programme we have measurements in place and
that we have measuring instruments in place for quality as well… every programme
must have qualitative and quantitative measuring instruments in place then we will be
able to balance out the [M&E] system and it would be better. (Participant 1, #1105)
In harmony with the recommendation suggested above, participants also proposed that the
agency could devise mechanisms to elicit feedback from the beneficiaries. By so doing, the
agency would have an indication of how it is doing as far as effective service delivery is
concerned:
152
I do agree entirely with the recommendations and findings... Is it not possible as an
organisation to design something like a template [to evaluate social work services],
which can be handed out to, [and completed by] our clients? (Participant 8, #1135)
In other words, participants highlighted the importance of adopting a participatory M&E
system in the agency, where the views of the beneficiaries are also considered for programme
improvement.
Given that recommendations for improvement of the M&E system (for the agency and for
child welfare in general) had also been solicited during the semi-structured interviews,
participants felt that these recommendations were vital in the improvement of the agency’s
M&E system. The recommendations that were generated at this stage as well as those that
were generated in the thematic discussion (during the focus group discussion and the semi-
structured interviews) are condensed and presented in the proceeding chapter.
6.11 Conclusion
Through the means of a focus group discussion, findings were thoroughly discussed and
clarified, and accordingly, opinions of the participants have been elicited. By and large,
participants found the findings to be a true reflection of the strengths and limitations of the
current M&E system of the agency. Findings were thus validated or augmented, and in some
instances disagreements were recorded or different interpretations were provided. Options for
improving the M&E system were explored. Recommendations for an appropriate and
effective M&E system for child welfare were generated thereby fulfilling the fourth objective
of the study.
The following chapter shall summarize the key findings of this study, make recommendations
and draw conclusions to this study.
153
Chapter Seven: Summary of Findings, Conclusions and
Recommendations
7.1 Introduction
Experience in rendering child protection services, familiarity with the collection and collation
of M&E data, and experience in supervising social workers in the child welfare field
provided the impetus for this study. The aim of the study was to evaluate the appropriateness
and effectiveness of the M&E system in a child welfare agency rendering community-based
services for OVCs and to make recommendations on how the system might be improved. In
order to achieve this goal, the student first conducted a situation analysis of the M&E system
in a specific child welfare agency. Second, an evaluation of what works and what does not
work in the implementation of the agency’s design and the utilization of the agency’s data
was conducted. Third, an evaluation of the utility and appropriateness of the existing M&E
data for organizational performance in rendering community-based services for OVCs was
carried out. Recommendations for an appropriate and effective M&E system for community-
based services for OVCs delivered in a child welfare setting were then made.
Whilst considering the importance of M&E in informing service delivery improvement in the
child welfare field, a lack of overall M&E systems and guidelines to assess the progress made
with the implementation of community-based care policies for OVCs was brought to light.
Such a striking situation is not unique to the child welfare sector but to the welfare field as a
whole (Patel et al., 2011). Therefore, the student’s intention was to use a solid research
methodology that could bring out findings which could contribute towards the development
of a more appropriate and effective M&E system for community-based services for OVCs.
Chapter 2 of this study highlighted the contextual background and the theoretical framework
for the development of an appropriate M&E system for organisations rendering community-
based services for OVCs. The contextual framework informing the study was a combination
of a theory-based approach (that is, the provision of generalist, multidisciplinary,
empowering, family-based and community-based services) and participatory M&E principles
and ideas (that is, active involvement of key stakeholders in the M&E process and joint
154
decision making). In addition, the study was guided by the utilization focused approach
whereby M&E forms part of the management tools of the agency in that it informs
programme design, intervention strategies and mechanisms for the fulfilment of goals,
objectives and plans.
The research method for this study was introduced and discussed in Chapter 3. Justification
was given as to why this study could be explored using a qualitative approach and a case
study design.
Chapter 4 presented and discussed the findings which emanated from the analysis of relevant
M&E documents obtained from the organisation where the study took place. This chapter
constituted the first phase of data collection.
Chapter 5 illuminated the findings that were elicited from the semi-structured interviews
(second phase of data collection). These findings answered questions and clarified issues
distilled from the findings obtained in Chapter 4 (first phase of data collection).
Chapter 6 presented the findings that were obtained from the focus group discussion (third
phase of data collection). In that focus group discussion, the student presented the findings
gathered thus far, explored participants’ perceptions of the findings and discussed options for
improving the M&E system. Recommendations for an appropriate and effective M&E system
for the child welfare field were also generated.
Section 7.2 below gives a succinct presentation of the key findings that emanated from the
study in terms of the current M&E situation (as is) in the agency and how the M&E system
operates in practice (what works and what does not work).
7.2 Summary of Key Findings
The following is a brief discussion of the ‘as is’ M&E system within the child welfare setting,
including the gaps that have been identified. The utility and appropriateness of the existing
M&E data for organisational performance in rendering community-based services for OVCs
is also considered. Key findings are, in this discourse, presented with respect to overall M&E
155
policy and guidelines; organisational structure; knowledge of the principles governing
community-based services for OVCs; M&E tools and processes within the agency; M&E
data processing, analysis and use; reporting; and training.
7.2.1 M&E System for OVCs in a Child Welfare Setting: Current Situation and Gaps
The situation analysis answered two key questions namely (a) what is the current M&E
system in the agency including the gaps; and (b) what works and what does not work in the
implementation of the agency’s M&E design and the utilization of the agency’s data. This
was based on a documentary study, the findings of which were set out in Chapter 4. Semi-
structured interviews conducted in Chapter 5 also provided insight which helped answer the
aforementioned questions. The following key findings are summarised with reference to these
two objectives of the study.
7.2.1.1 Overall M&E Policy and Guidelines
There appears to be no overall policy and guidelines for M&E in the agency. Rather,
the agency relies on the guidelines provided by the DSD as captured in the annual
business plan.
Since the agency has no clear M&E guidelines of its own, it has appropriately
structured and aligned its OVC programmes, plans, goals and objectives as required
by the DSD with its business plan.
Given the situation that there are no clear guidelines as stated above, there are also no
clearly crafted M&E plans to give direction on how OVC programmes ought to be
implemented and assessed.
Furthermore, there are no budgets allocated for M&E activities. All funds received by
the agency are allocated towards service delivery.
There are no overall indicators that the agency is tracking. Each specialised unit
(within the agency) has a set of objectives to achieve and under these objectives are
certain indicators, which show whether the objective is being achieved.
The six indicators that the agency is reportedly tracking (namely, distribution of
services, quality of services, quantity of services, time frame indicators, accessibility,
and issue and response specific indicators) are not documented. They are implicit,
rather than being clearly written down and tracked by the agency. There are no
156
documents in the agency that outline these indicators and that shows how these are
tracked over time.
7.2.1.2 Organisational Structure
The agency has a bureaucratic structure, and is divided into several specialised units
which are headed by a manager. Each unit is administered independently. As such,
there is little or no cooperation between units to achieve set objectives.
There is neither an M&E unit nor expert staff within the agency that are dedicated to
perform monitoring and evaluation. As a result, the M&E function is embedded in the
programmes of the agency and unit managers are tasked with the overall duty for
M&E.
The agency relies on co-ordination of M&E activities whereby, using a top-down
approach, a unit manager gives detailed and specific tasks through occasional ad hoc
meetings. M&E data is then collected by the social workers and submitted to their
supervisors where it is collated by the respective managers of each of the operating
units.
The described arrangement of the agency does not necessarily allow for smooth
referral of services and transfer of files between units. This is often time consuming
due to bureaucratic processes and protocols.
7.2.1.3 Knowledge of the Principles Governing Community-Based Services for OVCs
There is inadequate knowledge of the developmental approach in child welfare as the
majority of participants failed to operationalise this approach within child welfare.
Only middle line management demonstrated satisfactory knowledge of the principles
and fundamentals of the developmental approach.
The study found out that there is a fair understanding of the concept of M&E among
practitioners in the agency, notwithstanding the fact that M&E is regarded as a
preserve of managers. Supervisors and social workers equally exhibited an adequate
understanding of M&E. However, practitioners viewed M&E as, primarily, a funding
strategy and an accountability measure. They however acknowledged its importance
as a tool in promoting service improvements and quality.
The study also showed a general lack of understanding of what indicators are, which
was especially prevalent among social workers. Indicators were confused for goals or
157
trends. Similarly, participants also confused key concepts such as outputs with
outcomes.
7.2.1.4 Community-Based Services
Currently, there is a wide range of child protection programmes and community-based
services rendered for OVCs by the agency.
Nonetheless, there are no specific OVC programmes that offer home and community-
based care or deal with issues of poverty reduction, economic development as well as
social support and empowerment.
The current situation is that clients are either referred to other service providers for
such services or they are provided with piecemeal services that have been instituted to
cover this gap.
7.2.1.5 M&E Tools and Processes within the Agency
As highlighted above, M&E planning is regarded as a preserve of managers. This
implies that practitioners involved in the collection and collation of data hardly
participate in the determination of the overall goal, objectives, targets and plans.
Social workers perceive their M&E role to be peripheral. Resultantly, there is a
negative attitude towards the collection of M&E data, which is exacerbated by a lack
of understanding of the purpose and use of the data generated by the agency.
With regards to the design and implementation of the tools used by the agency to
collect M&E data for external purposes, there is neither participation nor training of
the practitioners in the design and implementation of the tools. The tools are imposed
on the agency as apparatuses to collect mandatory ‘statistics’ used for accountability
and justification of funding.
It was established through the study that tools used to collect M&E data for external
use are not well designed to elicit information which is sound and free from
subjectivity. Resultantly, data generated cannot be relied on since it is based on an
individual’s interpretation and understanding of what is required.
The organisational tools that are used for M&E data collection (for internal purposes)
and planning for service delivery are outdated, inadequate and underutilised. Some
are irrelevant to local context. Other tools require that the agency purchases licensing
rights in order to use them, which involves additional costs.
158
The agency has recently adopted an electronic M&E tool (SCS). However, there are
no guidelines or procedure manuals - or important information to inform staff on how
M&E data should be managed and used to inform assessment of programme
performance.
The SCS was not developed for the child welfare sector. As such, it has to be adapted
to meet the needs and address the challenges peculiar to child protection services.
There is unequal commitment to the use of the SCS between different units and
among practitioners within the agency. Thus, there is no optimal use of this M&E
system and data collection processes continue to be manual, labour intensive and
repetitive.
7.2.1.6 M&E Data Processing, Analysis and Use
There is no uniformity across the agency, first, of the tools used to collect M&E data
and, second, of the design of the tools used to collect M&E information. As such,
different types of data are collected, which presents challenges in the processing of
M&E data.
Moreover, there are no strict measures or systems to ensure that data collected is
correct. As such, it is difficult for the agency to ensure data integrity and quality.
The agency collects, through social workers, quantitative data related to inputs,
processes and outputs, which is then collated at the supervisor and managerial levels.
No qualitative data is collected by the agency.
The quantitative data is important to the agency for reporting to the DSD. However,
analysis required to translate such raw quantitative data to information on the
outcomes and impact of service delivery is complicated and thus lacking in the
agency.
Additionally, there is no tool in the agency that can be used to analyse voluminous
complex quantitative data. As a result, there is no data analysis conducted.
The ‘statistical’ information generated by the agency for internal and external
purposes is not well understood in terms of how it is used and its adequacy. There is
no evidence of the use of data for service delivery improvement within the agency.
159
7.2.1.7 Reporting
The main reporting structures of the agency are the six-monthly progress reports and
the annual report. Six-monthly progress reports are generated by the agency as they
are requirements stipulated by the DSD.
The SCS is also able to generate reports but there is no uniformity, currently, in the
use of this system to generate reports that can be used for internal and external
purposes.
M&E is not a topical issue in these reports and there is a lack of emphasis on the
M&E of community-based services for OVCs.
Through these aforementioned reports, the agency is able to report on output
indicators but cannot transcend this level of reporting to incorporate outcomes and
impacts.
Reporting seems to be one way. First, within the agency, M&E data is relayed from
social workers and supervisors to managers. Social workers and supervisors do not
get feedback on the M&E results. As a result, they remain unaware of the use of M&E
data they provide and the extent to which their services are making a difference to
beneficiaries. Second, between the agency and DSD, information goes from the
agency to the DSD. The agency hardly gets feedback from the DSD of the
information they provide on a monthly basis (through ‘statistics’) and over six months
(through the six-monthly progress reports). As such, the agency has no way of
measuring the extent to which they are achieving their objectives.
With the exception of the six-monthly progress reports, the agency has no other
reports for internal purposes that report on the progress being made in the
implementation of policies for OVCs and the effect of the services on the
beneficiaries. Therefore, the agency has no way of knowing what progress they are
making.
The M&E processes conducted at the supervisor level are not recorded and reported
on, which is a gap in the agency’s M&E system.
7.2.1.8 Training
Training workshops conducted by the agency are geared towards continuous
professional development, that is, to inculcate social work skills and to enhance
160
service delivery. Training received is not inclined towards providing staff responsible
for implementing M&E with guidance for continuous data quality improvement.
There is a significant lack of M&E skills among practitioners within the agency.
Managers and supervisors have, in the past, been trained in M&E. However, M&E
theory did not adequately equip them for practical M&E functioning.
7.2.2 M&E System for OVCs in a Child Welfare Setting: Evaluation of the M&E
Utility and Appropriateness
Findings in this section answered the second and third objective of the study, namely, to
establish what works and what does not work in the implementation of the agency’s design
and the utilization of the agency’s data; and to establish the utility and appropriateness of the
existing M&E data for organisational performance in rendering community-based services
for OVCs. Key findings that emanated from the documentary study, semi-structured
interviews as well as the focus group discussion to respond to these objectives are highlighted
below:
7.2.2.1 Overall M&E Policy and Guidelines
As highlighted earlier in Section 7.2.1, the agency has no overall policy, guidelines and
indicators, which give direction on how community-based services for OVCs ought to be
monitored and evaluated. The annual business plan is, however, instrumental in providing
guidance on what objectives to achieve and targets to meet. The annual business plan gives
the agency the basis to plan and the footing to measure progress. Nonetheless, the annual
business plan is rigid in structure. It does not allow room for the agency to incorporate other
programmes and capture other forms of data that could be valuable to the agency in response
to the dynamic issues dealt with in the child welfare field.
7.2.2.2 Organisational Structure
There seems to be a fit between the structure and the purpose of the agency. The
organisational structure is suitable and appropriate for the type of services (specialised)
rendered by the agency. The current arrangement is useful in the agency’s delivery of
services to clients and enables the agency to realise its objectives. However, as regards the
M&E function, the current organisational arrangement is problematic in that M&E is not
prioritised and sufficiently mainstreamed in the life of the agency. The child welfare field
161
requires technical M&E expertise. Thus, embedding the M&E function into the structures of
the agency leaving managers to play the central M&E role (as is the case currently in the
agency under study) poses a difficulty. M&E is not being regarded as a priority for the
agency.
7.2.2.3 Knowledge of the Principles Governing Community-Based Services for OVCs
In as much as practitioners within child welfare have the requisite social work knowledge and
skills, the introduction of new policies and approaches to child welfare has resulted in a
knowledge gap. Practitioners therefore need to acquaint themselves with the principles of
developmental child welfare. The agency has a good platform already set for educating and
providing guidance to practitioners (workshops, information sharing and training sessions),
which could be utilised to bridge the cited knowledge gap.
7.2.2.4 Community-Based Services
In light of the funding received by the agency, resources and the budgets allocated for OVC
programmes, the agency is able to appropriately respond to the OVC needs. It is, however,
constrained by resources (financial, material and infrastructural) hence the agency finds it
hard to expand its programmes and implement new initiatives.
7.2.2.5 M&E Tools and Processes in the Agency
The newly adopted SCS appears to be an appropriate tool aimed at alleviating the manual,
repetitive and tedious nature of M&E data collection in the agency. The SCS provides an
electronic platform where M&E processes can be conducted and M&E data generated.
However, tailoring the SCS is a time consuming process. The agency could have obtained,
from the onset, an M&E tool designed specifically to meet the needs in child welfare.
7.2.2.6 M&E Data Processing, Analysis and Use
The agency generates only quantitative information. Whilst this enables the agency to report
on output indicators as required by DSD, the collection and analysis of both quantitative and
qualitative information is more consistent with the child welfare field since that would
capture the quantifiable and intangible aspects of child welfare services. Moreover, the
collection of quantitative and qualitative data would enable the agency to evaluate progress
being made in implementing policies for OVCs and approaches to child welfare.
162
7.2.2.7 Reporting
The reporting structure used by the agency to report on progress made in light of the plans
enunciated in the annual business plan is a fairly straightforward process that works for the
agency. Gaps, however, exist in that there are no reports produced for internal purposes, that
is, to give feedback to practitioners on progress made with regard to set goals and objectives.
M&E reports could be produced more frequently for internal purposes, that is, to measure
progress in implementing OVC policies and to inform programme improvement.
The above discussion has highlighted what works and what does not work within the agency.
It has also highlighted the extent to which the current M&E system is useful and appropriate.
On the whole, Section 7.2 has created a picture of the current M&E system and gaps. In the
light of this, the following section will draw conclusions and discuss implications of these
findings for child welfare practice as a whole.
7.3 Conclusions
In light of the findings highlighted above, the following conclusions can be drawn pertaining
to M&E policy and practice, M&E implementation at agency level and further research.
The fact that there is no overall policy to govern how the implementation of community-
based services for OVCs should be monitored and evaluated, implies that the child welfare
field is operating without rules and principles to guide decision making (Rabie, 2011). This
challenge emanates from the fact that there is no overall national policy on M&E in the
welfare field (Patel et al., 2011). It is therefore difficult for the agency to achieve rational
outcomes. Moreover, the agency lacks baseline information to inform planning due to the fact
that there are no coherent procedures for M&E practice in child welfare (Schmid, 2012). It is
therefore an arduous task for South Africa to credibly comment on the progress being made
in the implementation of OVC policies and legislation.
Child welfare services have incalculable and intangible characteristics. As such, specific
indicators are required to give concrete but indirect values to particular concepts (Rabie,
2011). The finding that there are no overall indicators implies that child welfare is unable to
track organisational performance with respect to the implementation of OVC policies and
163
plans. Against this background, information on outcomes and impacts is therefore difficult to
draw on a national level.
Rabie and Cloete (2011) contend that organisations should have an M&E unit to provide
sound technical capacity for data collection and analysis. Also, the rule of thumb is that M&E
activities should be allocated between 5% and 10% of the project budget (USAID, 2010). The
child welfare agency under study, however, is operating without an M&E unit and there are
no budget allocations for M&E. It is therefore logical to conclude that there is a lack of
emphasis on M&E in the child welfare agency and it is impractical to expect the agency to
engage in M&E without appropriate technical expertise and adequate financial backup. The
agency is able to report on outputs but has no M&E capacity, adequate tools and appropriate
systems and infrastructure to measure outcomes and impacts. Thus, government-wide M&E
systems and national frameworks devised to spur M&E efforts in the public sector should
first consider capacitating organisations before policy implementation.
The study realised an unprecedented concern over government functioning and the influence
it exerts on the agency under study, since it is the main funder. There seems to be high
demands for accountability by the DSD, which detracts from the agency’s commitment to
effective service delivery (Schmid, 2012) and evaluation thereof. There seems to be poor
communication between the child welfare agency and the DSD. Information is provided to
the DSD but the NPO in this study hardly receives feedback, which appears to be a national
problem (Patel & Hochfeld, 2008). Data collection tools appear to be imposed on the agency
and changes are often effected without giving sufficient prior notice. The conclusion that can
be drawn from this finding is that the participatory approach to M&E, which informs this
study, has not been adopted wholly since there is no equal involvement, empowerment and
adherence to principles of mutual respect among stakeholders (Mathison, 2005). The
government could enhance ownership, participation, shared understanding and properly align
its policies by adopting PM&E principles and practices and creating an enabling environment
for stakeholders to discuss M&E policy directions and tools used in generating M&E data.
The study found out that the agency provides quantitative M&E data to the DSD on a
monthly basis. It is not understood how adequate this data is and what it is used for since
there is no analysis that is conducted of the data (by the agency) and no feedback (of analysis,
if any) is given to the agency by the DSD. The conclusion that can be drawn from this finding
164
is that utilization-focused principles have not yet been fully incorporated since M&E data is
not being used to inform programme design, service delivery improvement, and to devise
appropriate strategies for goal achievement (Patton, 2004). Data currently being generated
could be used optimally to inform policy makers and implementing agencies on the
appropriateness of OVC policies and approaches.
Rabie and Cloete (2011) suggest that M&E has a role to play in establishing why, where, and
for whom programmes work or fail. In other words, M&E plays a pivotal role in evaluating
theories that inform child welfare practice. The finding that there is a poorly constructed
M&E system within child welfare agency is concerning. It implies that the child welfare
agency has no sound capacity to evaluate the extent to which principles and practices of the
developmental approach and more specifically, the developmental child welfare model
(Schmid, 2012) are being implemented. One may infer from this finding that there is a
difficulty being experienced by child welfare NGOs and the government in commenting on
the appropriateness and effectiveness of child welfare policies and theories on a national
scale.
On the whole, this study has portrayed that the agency under study has a poorly constructed
M&E system owing to a lack of overall policy and indicators. This is a major gap that
continues to exist despite the tailored solution being devised by the agency. Kusek and Rist
(2004) assert that results-based M&E systems are valuable in informing organisations of the
worth of their programmes. Mathison (2005) suggests that M&E systems should be
participatory in nature and that they should form part of the organisational management tool
(Patton, 2004). Such M&E systems that are results-based, participatory and utilization
focused are of immense significance to the child welfare field as they will provide valuable
information on the implementation of the developmental approach for OVCs. The World
Bank (2013) asserts that the welfare field needs to concentrate and intensify its efforts on
M&E in order to realise the outcomes and the impact of government spending. It is against
this backdrop that the student makes the following recommendations, first, for M&E policy
and practice, then M&E implementation at agency level, and, finally, for further research.
165
7.4 Recommendations for M&E Policy and Practice
The following are recommendations directed to the government/policy makers. There is an
overall need to:
develop M&E policies to guide how community-based services for OVCs are to be
monitored and evaluated on the whole;
build capacity and ensure that appropriate training is offered for the implementation
of M&E policies within the child welfare field;
develop a child welfare M&E system that provides the basis for M&E planning;
improve communication between the DSD (funder/ policy maker) and the child
welfare agency (policy implementer) through, for example, giving timely notice to
NGOs before effecting changes in tools or data requirements;
adopt participatory approaches in the design of data collection tools and strive for
uniformity in the tools and data required on a monthly basis;
encourage the development of more programmes and establishment of new initiatives
for OVCs with proper and adequate provision of funds and other resources to
capacitate practitioners for effective service delivery;
allocate budgets for the child welfare agency to specifically finance the budget for
M&E;
train social workers and social service professionals in evaluation research so that
there can be a shift towards the evaluation of outcomes and impacts in the child
welfare field;
align social work education so that it equips students with skills to effectively execute
monitoring and evaluation duties in the workplace.
7.5 Recommendations for M&E Implementation at Agency Level
In view of the findings generated and the discussion above of the conclusions that have been
drawn from this study, it is recommended for the agency to:
draft internal M&E policies to govern how services should be monitored and
evaluated;
develop overall indicators that can be tracked (by the agency) to measure overall
performance and report on outcomes and impact;
166
establish an M&E unit with proper funding to provide the sound technical support
required;
invest in M&E training of staff at all levels to create fundamental knowledge of
M&E;
adopt a participatory approach in the determination of objectives and indicators;
design more programmes and come up with initiatives that are specifically targeted to
foster economic development and poverty reduction;
conduct regular information sharing sessions between managers and the staff to share
information on the overall goal, objectives and indicators contained in the business
plans;
invest in staff training on what is developmental child welfare practice and how to
contextualise and operationalise the developmental approach in child welfare;
collect feedback from the service users on the effectiveness of the services being
rendered;
channel resources towards a review of current M&E tools and the development of
context specific and up-to-date tools to measure the qualitative aspects of service
delivery that meets the specific needs of the agency and of their clients;
invest more in the internal M&E system (SCS) so that it is less labour intensive or
complex, and that there is internal policy on how the SCS is to be used as an M&E
system;
utilise the SCS more optimally and encourage equal commitment to the system from
the different units and practitioners within the agency;
enhance cohesion between units so that there is cooperation in achieving the overall
goal, meeting targets and realising objectives;
seek more resources (financial, infrastructural and so on) to enable practitioners to
execute their M&E duties more effectively.
The above recommendations are pertinent to other child welfare agencies that are grappling
with how to give effect to new welfare policies and how to monitor and evaluate these
policies.
167
7.6 Recommendations for Further Research
Considering the research processes and the findings of this research, the following
recommendations are made for further research:
There is an immediate need for further research to be conducted with the view to
developing an M&E framework for the child welfare field.
Further research could also be conducted to explore how community-based services
for OVCs are being rendered from a developmental perspective and evaluate the
extent to which OVC policies and approaches are being successfully implemented.
Further research could also involve the service users (clients/beneficiaries) in the
collection of data as well as to find out their views on how services ought to be
monitored and evaluated, and the kind of data they think should be collected by child
welfare agencies. It could also be conducted using a large sample representative of all
the units within the agency
Further research could actually conduct a quantitative and qualitative analysis of data
collected by the agency to demonstrate the kind of results that M&E data currently
being collected could generate.
7.7 Final Word
This qualitative study has explored the M&E system as it exists within a specific child
welfare agency. Relevant data, which was triangulated with theory, to suggest the ‘what
could be’ M&E situation was elicited from the agency M&E documents and participants.
Thus, the approach, methods, as well as the data collection tools utilised in this study were
effective in addressing the goal and objectives of this study. Data analysis was conducted
appropriately, leading to the generation of findings, conclusions and recommendations as
discussed in this chapter.
An appropriate and effective M&E system for child welfare needs to be devised. Indicators
need to be developed and, additionally, M&E needs to be prioritised. However, this study has
endeavoured to investigate and put forward recommendations that will perhaps viably
address these challenges.
168
References
Ackron, J. (2013). Public sector monitoring and evaluation. Unpublished manuscript,
University of Stellenbosch, Cape Town, South Africa.
Andersson, G. (2003). Evaluation of the contract family service in Sweden. In I. Katz & J.
Pinkerton (Eds.), Evaluating family support: Thinking internationally, thinking
critically (pp. 291-306). West Sussex, England: John Wiley and Sons.
Ansell, N. (2005). Children, youth and development. New York, USA: Routledge.
Babbie, E. (2001). The practice of social science research. Belmont, CA: Wadsworth.
Bless, C., & Higson-Smith, C. (1995). Fundamentals in social research methods: An African
perspective (2nd ed.). Kenwyn, RSA: Juta and Co.
Burman, S. (2003). The best interests of the South African child. International Journal of
Law, Policy and the Family. 17(1), 28-40.
Burns, N., & Grove, S. K. (2009). The practice of nursing research: Appraisal, synthesis and
generation of evidence (6th ed.). Philadelphia, PA: Saunders/Elsevier.
Brendtro, L. K., & Larson, S. J. (2006). The resilience revolution: Discovering strengths in
challenging kids. Bloomington, IN: Solution Tree Press.
Cape Town Child Welfare. (2013). Community family strengthening. Retrieved May 12,
2013, from
http://www.childwelfaresa.org.za/cp1.html
Child Welfare Information Gateway. (2013). Guiding principles of systems of care:
Community-based services. Retrieved February 20, 2013, from
https://www.childwelfare.gov/management/reform/soc/history/community.cfm
169
Children’s Act, Act No. 38 of 2005. Pretoria, RSA: Government Printers.
Child Welfare South Africa. (2013). Interventions for orphans and vulnerable children at
four project sites in South Africa. Cape Town, RSA: HSRC Press.
Constitution of the Republic of South Africa, Act No. 108 of 1996. Pretoria, RSA:
Government Printers. Retrieved September 24, 2012, from
http://www.info.gov.za/documents/constitution/1996/a108-96.pdf
Creswell, J. W. (2009). Research design: Qualitative, quantitative and mixed method
approaches (3rd ed.). Thousand Oaks, CA: SAGE Publications.
Department of Social Development. (1997). White paper for social welfare. Pretoria, RSA:
Government Printers.
Department of Social Development. (2006a). National action plan for orphans and other
children made vulnerable by HIV and AIDS South Africa 2006-2008. Pretoria, RSA:
Government Printers.
Department of Social Development. (2006b). Policy framework for orphans and other
children made vulnerable by HIV and AIDS, South Africa. Pretoria, RSA: Government
Printers. Retrieved April 17, 2013, from
http://www.cindi.org.za/files/Policy-Framework-for-OVC-Final.pdf
Department of Social Development. (2009). National action plan for orphans and other
children made vulnerable by HIV and AIDS South Africa 2009-2012. Pretoria, RSA:
Government Printers.
Department of Social Development. (2010). Consolidated regulations pertaining to the
Children’s Act, 2005. Pretoria, RSA: Government Printers.
Department of Social Development. (2011). Framework for provision of developmental
social welfare services. Pretoria, RSA: Government Printers.
170
Department of Women, Children and People with Disabilities (2012a). National action plan
for orphans and other children made vulnerable by HIV and AIDS South Africa 2012-
2017. Pretoria, RSA: Government Printers.
Department of Women, Children and People with Disabilities. (2012b). National plan of
action for children in South Africa. Pretoria, RSA: Government Printers.
De Vos, A. S., Strydom, H., Fouché, D. R., & Delport, C. S. L. (Eds.). (2011). Research at
grassroots (4th ed.). Pretoria, RSA: Van Schaik Publishers.
Dutschke, M. (2007). Defining children’s constitutional right to social services. Cape Town,
RSA: University of Cape Town, Children’s Institute.
Fluke, J. D., & Wulczyn, F. (2010). A concept note on child protection systems monitoring
and evaluation. Chicago, IL: Chapin Hall.
Fouché, C. B., & Delport, C. S. L. (2011). Introduction to the research process. In A. S. De
Vos, H. Strydom, C. B. Fouché, & C. S. L. Delport (Eds.), Research at grass roots: For
the social sciences and human service professions (4th ed., pp. 61-88). Pretoria, RSA:
Van Schaik Publishers.
Freymond, N., & Cameron, G. (Eds.). (2006). Towards positive systems of child and family
welfare: International comparisons of child protection, family service and community
caring systems. Toronto, Canada: University of Toronto Press.
Gilbert, N., Parton, N., & Skivenes, M. (2011). Child protection systems: International trends
and orientations. New York, USA: Oxford University Press.
Gray, M. (2008). The progress of social development in South Africa. International Journal
of Social Welfare, 15, 53-64.
Greeff, M. (2011). Information collection: Interviewing. In A. S. De Vos, H. Strydom, C. B.
Fouché, & C. S. L. Delport (Eds.), Research at grass roots: For the social sciences and
human service professions (4th ed., pp. 341-375). Pretoria, RSA: Van Schaik Publishers.
171
Green, S. (2008). Perspectives of some non-governmental organizations on progress towards
developmental social welfare and social work. The Social Work Practitioner-
Researcher, 20(2), 174-191.
Greene, J. C. (2006). Evaluation, democracy, and social change. In I. F. Shaw, J. C. Greene,
& M. M. Mark (Eds.), The SAGE handbook of evaluation (2nd ed., pp. 124-132).
London, UK: SAGE Publications.
Guijt, I., & Gaventa, J. (1998). Participatory monitoring and evaluation: Learning from
change. IDS working paper, 12. University of Sussex, Brighton: IDS.
Hochfeld, T., & Selipsky, L. (2011). Towards the design of social welfare indicators for
welfare NGOs: Work session report. Johannesburg, RSA: Centre for Social
Development in Africa, University of Johannesburg.
Hölscher, D. (2008). The emperor’s new clothes: South Africa’s attempted transition to
developmental social welfare and social work. International Journal of Social Welfare,
17(2), 114-123.
Imas, L. G., & Rist, R. C. (2009). The road to results: Designing and conducting effective
development evaluations. Washington, DC: The World Bank.
Kadushin, A., & Harkness, D. (2002). Supervision in social work. Columbia, SC: Columbia
University Press.
Katz, I., & Pinkerton, J. (Eds.). (2003). Evaluating family support: Thinking internationally,
thinking critically. West Sussex, England: John Wiley and Sons.
Kirst-Ashman, K. K. (2007). Introduction to social work and social welfare: Critical thinking
perspectives (2nd ed.). Belmont, CA: Thomson Brooks/Cole.
Kusek, J. Z., & Rist, R. C. (2004). Ten steps to a results-based monitoring and evaluation
system. Washington, DC: The World Bank.
172
Lawrence, C., Strolin-Goltzman, J., Caringi, J., Claiborne, N., McCarthy, M., Butts, E., &
O’Connell, K. (2013). Designing evaluations in child welfare organizations: An
approach for administrators. Administration in Social Work, 37, 3-13.
Leedy, P. D. (2010). Practical research: Planning and design. Upper Saddle River, NJ:
Merrill.
Lewis, J., Lewis, M., Packard, T., & Souflee, F. (2001). Management of human service
programs (3rd ed.). Belmont, CA: Brooks/Cole.
Loffell, J. (2000). Truth and the social services. Social Work/Maatskaplike Werk, 36(1), 53-
68.
Loffell, J. (2008). Developmental social welfare and the child protection challenge in South
Africa. Practice: Social Work in Action, 20(2), 83-91.
Lombard, A. (2007). The impact of social welfare policies on social development in South
Africa: An NGO perspective. Social Work/Maatskaplike Werk, 43(4), 295-316.
Mark, M. M., Greene J. C., & Shaw, I. F. (2006). Introduction: The evaluation of policies,
programs and practices. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The SAGE
handbook of evaluation (3rd ed., pp. 6-21). London, UK: SAGE Publications.
Martin, P. (2003). Against management: Organization in the age of managerialism. Oxford,
UK: Blackwell.
Mathews, S., Jamieson, L., Lake, L., & Smith, C. (Eds.). (2014) South African child gauge.
Cape Town, RSA: Children’s Institute, University of Cape Town.
Mathison, S. (Ed.). (2005). Encyclopaedia of evaluation (2nd ed.). Thousand Oaks, CA:
SAGE Publications.
173
Midgley, J. (2001). Strategies for social development: The developmental perspective in
social welfare. London, UK: SAGE Publications.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded source
book (2nd ed.). Thousand Oaks, CA: SAGE Publications.
National Treasury. (2007). Framework for managing programme performance information
(RP:86/2007). Pretoria, RSA: Formeset Printers Cape.
Neuman, W. L. (2000). Social research methods: Qualitative and quantitative approaches
(3rd ed.). Boston, MA: Allyn & Bacon.
Noyoo, N. (2005). Monitoring and evaluation for social development: A case for social work
in South Africa. Social Work/Maatskaplike Werk, 41(3), 229-236.
Nyangara, F., Kalungwa, Z., Obiero, W., Thurman, T. R., & Chapman, J. (2009). Promoting
a home-based program model for supporting children affected by HIV/AIDS:
Evaluation of Tumaini project in Iringa region, Tanzania. Iringa, Tanzania: USAID.
Patel, L. (2005). Social welfare and social development in South Africa. Cape Town, RSA:
Oxford University Press.
Patel, L. (2008). Getting it right and wrong: An overview of a decade of post-apartheid social
welfare. Practice: Social Work in Action, 20(2), 71-81.
Patel, L. (in press). Social welfare and social development in South Africa.
Patel, L., Grobbelaar, M., Selipsky, L., & Curtis, K. (2011). Towards the design of social
welfare indicators for welfare NPOs: A concept paper and indicator framework.
Johannesburg, RSA: Centre for Social Development in Africa, University of
Johannesburg.
174
Patel, L., & Hochfeld, T. (2008). Indicators, barriers and strategies to accelerate the pace of
change to developmental welfare in South Africa. The Social Work Practitioner-
Researcher 20(2), 192-211.
Patel, L., & Hochfeld, T. (2011). Doornkop survey on gender and child support grants:
Preliminary results. Johannesburg, RSA: Centre for Social Development in Africa,
University of Johannesburg.
Patel, L., Schmid, J., & Hochfeld, T. (2012). Transforming social work services in South
Africa: Perspectives of NPO managers. Administration in Social Work, 36, 212-230.
Patton, M. Q. (2004). The roots of utilization-focused evaluation. In C. M. Alkin (Ed.),
Evaluation roots: Tracing theorists’ views and influences (3rd ed., pp. 265-298).
Thousand Oaks, CA: SAGE Publications.
Rabie, B. (2011). Public sector monitoring and evaluation. Unpublished manuscript,
University of Stellenbosch, Cape Town, South Africa.
Rabie, B., & Cloete, F. (2011). Policy evaluation. In F. Cloete & C. de Coning (Eds.),
Improving public policy: Theory, practice and results (6th ed., pp. 196-214). Pretoria,
RSA: Van Schaik Publishers.
Rabie, B., & Cloete, F. (n.d.). Re-classification of M&E approaches. Johannesburg, RSA:
School of Public Management and Planning, University of Stellenbosch and
Department of Public Governance, University of Johannesburg.
Reynecke, J. H., Steyn, M. M., & Rankin, P. (2007) Evaluation of volunteers' general
experience of the Isolabantwana/Eye on the Children's project. Journal for
Psychosocial Behavioural Science, 20(1), 21-26.
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach
(7th ed.). London, UK: SAGE Publications.
175
Rubin, A., & Babbie, E. (2011). Research methods for social work (7th ed.). Belmont, CA:
Cengage Learning.
Schenk, K. D. (2009). Community interventions providing care and support to orphans and
vulnerable children: A review of evaluation evidence. AIDS Care, 21(7), 918-942.
Schmid, J. (2008). The story of South African child welfare: A history of the present
(Doctoral dissertation). Wilfrid Laurier University, Waterloo, Canada.
Schmid, J. (2009). Subjectivities in South African child welfare discourse. Transformation:
Critical Perspectives on Southern Africa, 70, 92-118.
Schmid, J. (2012). Child welfare and children’s mental health needs in the context of
development: An integrated (South African) approach. Child and Adolescent Mental
Health, 1, 200-215.
Schmid, J. (2013). Recounting stories in South African child welfare. Social
Work/Maatskaplike Werk, 49(3), 294-308.
Schurink, W., Fouché, C. B., & De Vos, A. S. (2011). Qualitative data analysis and
interpretation. In A. S. De Vos, H. Strydom, C. B. Fouché, & C. S. L. Delport (Eds.),
Research at grass roots: For the social sciences and human service professions (4th ed.,
pp. 397-423). Pretoria, RSA: Van Schaik Publishers.
Scott, J., & Marshall, G. (2005). Oxford dictionary of sociology. Oxford, UK: Oxford
University Press.
Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects.
Education for Information, 22, 63-75.
Sibanda, S. (2013). Challenges faced by social workers working in child protection services
in implementing the Children’s Act No. 38 of 2005 (Masters dissertation). University of
Pretoria, Pretoria, RSA.
176
Smart, R. (2003). Policies for orphans and vulnerable children: A framework for moving
ahead. Washington, DC: POLICY/USAID.
Statistics South Africa. (2010). Statistical release: Midyear population estimates. Pretoria,
RSA: Government Printers.
Struwig, F. W., & Stead, G. B. (2001). Planning, designing and reporting research. Cape
Town, RSA: Pearson Education.
Strydom, H. (2011a). Ethical aspects of research in the social sciences and human service
professions. In A. S. De Vos, H. Strydom, C. B. Fouché, & C. S. L. Delport (Eds.),
Research at grass roots: For the social sciences and human service professions (4th ed.,
pp. 113-130). Pretoria, RSA: Van Schaik Publishers.
Strydom, H. (2011b). The pilot study in the quantitative paradigm. In A. S. De Vos, H.
Strydom, C. B. Fouché, & C. S. L. Delport (Eds.), Research at grass roots: For the
social sciences and human service professions (4th ed., pp. 236-247). Pretoria, RSA:
Van Schaik Publishers.
Strydom, H., & Delport, C. S. L. (2011). Sampling and pilot study in qualitative research. In
A. S. De Vos, H. Strydom, C. B. Fouché, & C. S. L. Delport (Eds.), Research at grass
roots: For the social sciences and human service professions (4th ed., pp. 390-396).
Pretoria, RSA: Van Schaik Publishers.
Stufflebeam, D. L. (1999). Foundational models for 21st century program evaluation.
Kalamazoo, MI: The Evaluation Centre, Western Michigan University.
Stufflebeam, D. L. (2004). The 21st century CIPP model: Origins, development and use. In
Alkin, C. M. (Ed.), Evaluation roots: Tracing theorists’ views and influences (2nd ed.,
pp. 245-266). Thousand Oaks, CA: Sage Publications.
Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models and applications.
San Francisco, CA: Jossey-Bass.
177
Taylor, G. R. (2000). Introduction and overview of the research process. In G. R. Taylor
(Ed.), Integrating quantitative and qualitative methods in research (3rd ed., pp. 51-76).
Lanham, MD: University Press of America.
Terrell, S. (2011). Mixed-methods research methodologies. The Qualitative Report, 17(1),
254-280. Retrieved June 15, 2012, from
http://www.nova.edu/ssss/QR/QR17-1/terrell.pdf
Thiele, S. (2005). Exploring the feasibility of foster care as a primary permanency option for
orphans (Masters dissertation). University of South Africa, Pretoria, RSA.
Thurman, T. R., Hutchinson, P., Ikamari, L., Gichuhi, W., Murungaru, K., & Nyangara, F.
(2009). Community education and sensitization as an OVC care and support strategy:
Evaluation of the integrated AIDS program-Thika in Kenya. Nairobi, Kenya:
MEASURE/Evaluation.
Thurman, T. R., Snider, L., Boris, N., Kalisa, E., Nyirazinyoye, L., & Brown, L. (2008).
Barriers to the community support of orphans and vulnerable youth in Rwanda. Social
Science & Medicine, 66(7), 1557-1567.
UNAIDS. (2010). Monitoring and evaluation fundamentals: An introduction to indicators.
Geneva, Switzerland: UNAIDS.
UNICEF. (2009a). Going global with indicators of child well-being: Indicators of South
African children’s psychosocial development in the early childhood period
(ECD/2004/1/2774). Cape Town, RSA: HSRC.
UNICEF. (2009b). Guidance document: Developing and operationalizing a national
monitoring and evaluation system for the protection, care and support of orphans and
vulnerable children living in a world with HIV and AIDS. New York, USA: UNICEF.
UNICEF. (2009c). Situation analysis of children in South Africa. Pretoria, RSA: The
Presidency/UNICEF.
178
UNICEF. (2013). United Nations convention on the rights of the child. Retrieved March 14,
2013, from
http://www.uncrcletsgetitright.co.uk/index.php/uncrc
Waldegrave, C. (2006). Contrasting national jurisdictional and welfare responses to violence
to children. Social Policy Journal of New Zealand, 27, 57-76.
Weber, M. (1947). The theory of social and economic organisation. New York, USA: The
Free Press.
Wilson, T. (2008). Isibindi: A model for providing community-based child and youth care
services. The Social Work Practitioner-Researcher, 20(3), 407-415.
World Bank. (2002). Monitoring and evaluation: Some tools, methods and approaches.
Washington, DC: World Bank.
World Bank. (2004). Ten steps to building a results-based monitoring and evaluation system.
Washington, DC: World Bank.
World Bank. (2006). Orphans and vulnerable children core definitions. Retrieved February
19, 2012, from
http://info.worldbank.org/etools/docs/library/164047/howknow/definitions.htm
World Bank. (2013). Monitoring and evaluation. Retrieved July 28, 2013, from
http://info.worldbank.org/etools/docs/library/162495/howdo/monitoring.htm
Yin, R. K. (2004). Case study research: Design and method (3rd ed.). Thousand Oaks, CA:
SAGE Publications.
179
Annexures
Related Documents
