Evaluation: What is it and why do it?
Evaluation. What associations does this word bring to mind? Do
you see evaluation as an invaluable tool to improve your program?
Or do you find it intimidating because you don't know much about
it? Regardless of your perspective on evaluation, MEERA is here to
help! The purpose of this introductory section is to provide you
with some useful background information on evaluation.
Table of Contents What is evaluation?
Why should I evaluate my program?
What type of evaluation should I conduct and when?
What makes a good evaluation?
How do I make evaluation an integral part of my program?
How can I learn more?
What is evaluation?
Evaluation is a process that critically examines a program. It
involves collecting and analyzing information about a programs
activities, characteristics, and outcomes. Its purpose is to make
judgments about a program, to improve its effectiveness, and/or to
inform programming decisions (Patton, 1987).
Why Should I evaluate my program?
Experts stress that evaluation can:
There are some situations where evaluation may not be a good
idea!
when the program is unstable, unpredictable, and has no
consistent routine
when those involved cannot agree about what the program is
trying to achieve
when a funder or manager refuses to include important and
central issues in the evaluationAdapted from Thomson and Hoffman
(2003).
Improve program design and implementation.It is important to
periodically assess and adapt your activities to ensure they are as
effective as they can be. Evaluation can help you identify areas
for improvement and ultimately help you realize your goals more
efficiently. Additionally, when you share your results about what
was more and less effective, you help advance environmental
education.
Demonstrate program impact. Evaluation enables you to
demonstrate your programs success or progress. The information you
collect allows you to better communicate your program's impact to
others, which is critical for public relations, staff morale, and
attracting and retaining support from current and potential
funders.
Why conduct evaluations?, approx. 2 minutes
Gus Medina, Project Manager, Environmental Education and
Training Partnership
What type of evaluation should I conduct?
Make evaluation part of your program; dont tack it on at the
end!
The evaluation experience is likely to be more positive and its
results are likely to be more useful if you build evaluation in
from the start and make it an on-going activity. For more
information on this topic, click hereEvaluations fall into one of
two broad categories: formative and summative. Formative
evaluations are conducted during program development and
implementation and are useful if you want direction on how to best
achieve your goals or improve your program. Summative evaluations
should be completed once your programs are well established and
will tell you to what extent the program is achieving its
goals.
Within the categories of formative and summative, there are
different types of evaluation:
Type of EvaluationPurpose
Formative
1. Needs AssessmentDetermines who needs the program, how great
the need is, and what can be done to best meet the need. An EE
needs assessment can help determine what audiences are not
currently served by programs and provide insight into what
characteristics new programs should have to meet these audiences
needs.
For more information, Needs Assessment Training uses a practical
training module to lead you through a series of interactive pages
about needs assessment.
2. Process or Implementation EvaluationExamines the process of
implementing the program and determines whether the program is
operating as planned. Can be done continuously or as a one-time
assessment. Results are used to improve the program. A process
evaluation of an EE program may focus on the number and type of
participants reached and/or determining how satisfied these
individuals are with the program.
Summative
1. Outcome EvaluationInvestigates to what extent the program is
achieving its outcomes. These outcomes are the short-term and
medium-term changes in program participants that result directly
from the program. For example, EE outcome evaluations may examine
improvements in participants knowledge, skills, attitudes,
intentions, or behaviors.
2. Impact EvaluationDetermines any broader, longer-term changes
that have occurred as a result of the program. These impacts are
the net effects, typically on the entire school, community,
organization, society, or environment. EE impact evaluations may
focus on the educational, environmental quality, or human health
impacts of EE programs.
For additional information on the differences between outcomes
and impacts, including lists of potential EE outcomes and impacts,
see MEERA's Outcomes and Impacts page.
Which of these evaluations is most appropriate depends on the
stage of your program:
Adapted from Norland (2004), Pancer and Westhues (1989) and
Rossi et al. (2004).
What makes a good evaluation?
Consider doing a best practices review of your program before
proceeding with your evaluation.
If you are not clear on what you want to evaluate, consider
first reviewing your program based on best practices such as the
North American Association for Environmental Education (2004)
Guidelines for Excellence Series:1. Nonformal Environmental
Education Programs: Guidelines for Excellence
2. Environmental Education Materials: Guidelines for
Excellence
3. Excellence in Environmental Education: Guidelines for
Learning (preK-12)
4. Guidelines for the Preparation and Professional Development
of Environmental Educators
You can conduct this review yourself or you can ask another
environmental educator to help you. A best practices review is
likely to identify program strengths and weaknesses, giving you
important insight into what to focus your evaluation on.
A well-planned and carefully executed evaluation will reap more
benefits for all stakeholders than an evaluation that is thrown
together hastily and retrospectively. Though you may feel that you
lack the time, resources, and expertise to carry out an evaluation,
learning about evaluation early-on and planning carefully will help
you navigate the process.
MEERA provides suggestions for all phases of an evaluation. But
before you start, it will help to review the following
characteristics of a good evaluation (list adapted from resource
formerly available through the University of Sussex, Teaching and
Learning Development Unit Evaluation Guidelines and John W. Evans'
Short Course on Evaluation Basics):
Good evaluation is tailored to your program and builds on
existing evaluation knowledge and resources. Your evaluation should
be crafted to address the specific goals and objectives of your EE
program. However, it is likely that other environmental educators
have created and field-tested similar evaluation designs and
instruments. Rather than starting from scratch, looking at what
others have done can help you conduct a better evaluation. See
MEERAs searchable database of EE evaluations to get started.
Good evaluation is inclusive. It ensures that diverse viewpoints
are taken into account and that results are as complete and
unbiased as possible. Input should be sought from all of those
involved and affected by the evaluation such as students, parents,
teachers, program staff, or community members. One way to ensure
your evaluation is inclusive is by following the practice of
participatory evaluation.
Good evaluation is honest. Evaluation results are likely to
suggest that your program has strengths as well as limitations.
Your evaluation should not be a simple declaration of program
success or failure. Evidence that your EE program is not achieving
all of its ambitious objectives can be hard to swallow, but it can
also help you learn where to best put your limited resources.
Good evaluation is replicable and its methods are as rigorous as
circumstances allow. A good evaluation is one that is likely to be
replicable, meaning that someone else should be able to conduct the
same evaluation and get the same results. The higher the quality of
your evaluation design, its data collection methods and its data
analysis, the more accurate its conclusions and the more confident
others will be in its findings.
How do I make evaluation an integral part of my program?
Making evaluation an integral part of your program means
evaluation is a part of everything you do. You design your program
with evaluation in mind, collect data on an on-going basis, and use
these data to continuously improve your program.
Developing and implementing such an evaluation system has many
benefits including helping you to:
better understand your target audiences' needs and how to meet
these needs
design objectives that are more achievable and measurable
monitor progress toward objectives more effectively and
efficiently
learn more from evaluation
increase your program's productivity and effectiveness
To build and support an evaluation system:
Couple evaluation with strategic planning. As you set goals,
objectives, and a desired vision of the future for your program,
identify ways to measure these goals and objectives and how you
might collect, analyze, and use this information. This process will
help ensure that your objectives are measurable and that you are
collecting information that you will use. Strategic planning is
also a good time to create a list of questions you would like your
evaluation to answer.
Revisit and update your evaluation plan and logic model (See
Step 2) to make sure you are on track. Update these documents on a
regular basis, adding new strategies, changing unsuccessful
strategies, revising relationships in the model, and adding
unforeseen impacts of an activity (EMI, 2004).
Build an evaluation culture by rewarding participation in
evaluation, offering evaluation capacity building opportunities,
providing funding for evaluation, communicating a convincing and
unified purpose for evaluation, and celebrating evaluation
successes.
The following resource provides more depth on integrating
evaluation into program planning:
Best Practices Guide to Program Evaluation for Aquatic Educators
(.pdf)Recreational Boating and Fishing Foundation. (2006).Beginner
IntermediateChapter 2 of this guide, Create a climate for
evaluation, gives advice on how to fully institutionalize
evaluation into your organization. It describes features of an
organizational culture, and explains how to build teamwork,
administrative support and leadership for evaluation. It discusses
the importance of developing organizational capacity for
evaluation, linking evaluation to organizational planning and
performance reviews, and unexpected benefits of evaluation to
organizational culture.
If you want to learn more about how to institutionalize
evaluation, check out the following resources on adaptive
management. Adaptive management is an approach to conservation
management that is based on learning from systematic, on-going
monitoring and evaluation, and involves adapting and improving
programs based on the findings from monitoring and evaluation.
Adaptive Management: A Tool for Conservation
PractitionersSalafsky, N., R. Margoluis, and K. Redford, (2001)
Biodiversity Support Program. BeginnerThis guide provides an
overview of adaptive management, defines the approach, describes
the conditions under which adaptive managements makes most sense,
and outlines the steps involved.
Measures of Conservation Success: Designing, Managing, and
Monitoring Conservation and Development ProjectsMargoluis, R., and
N. Salafsky. (1998) Island Press.Beginner Intermediate
AdvancedAvailable for purchase at Amazon.com.This book provides a
detailed guide to project management and evaluation. The chapters
and case studies describe the process step-by-step, from project
conception to conclusion. The chapters on creating and implementing
a monitoring plan, and on using the information obtained to modify
the project are particularly useful.
How can I learn more?
Does your project make a difference? A guide to evaluating
environmental education projects and programs. Sydney: Department
of Environment and Conservation, Australia. (2004)BeginnerSection 1
provides a useful introduction to evaluation in EE. It defines
evaluation, and explains why it is important and challenging, with
quotes about the evaluation experiences of several environmental
educators.
Designing Evaluation for Education Projects (.pdf),NOAA Office
of Education and Sustainable Development. (2004)BeginnerIn Section
3, Why is evaluation important to project design and
implementation? nine benefits of evaluation are listed, including,
for example, the value of using evaluation results for public
relations and outreach.
Evaluating EE in Schools: A Practical Guide for Teachers
(.pdf),Bennett, D.B. (1984). UNESCO-UNEPBeginner IntermediateThe
introduction of this guide explains four main benefits of
evaluation in EE, including: 1) building greater support for your
program, 2) improving your program, 3) advancing student learning,
and 4) promoting better environmental outcomes.
Guidelines for Evaluating Non-Profit Communications Efforts
(.pdf),Communications Consortium Media Center. (2004)Beginner
IntermediateA section titled Overarching Evaluation Principles
describes twelve principles of evaluation, such as the importance
of being realistic about the potential impact of a project, and
being aware of how values shape evaluation. Another noteworthy
section, Acknowledging the Challenges of Evaluation, outlines nine
substantial challenges, including the difficulty in assessing
complicated changes in multiple levels of society (school,
community, state, etc.). This resource focuses on evaluating public
communications efforts, though most of the content is relevant to
EE.
References:EMI (Ecosystem Management Initiative). (2004).
Measuring Progress: An Evaluation Guide for Ecosystem and
Community-Based Projects. School of Natural Resources and
Environment, University of Michigan. Downloaded September 20, 2006
from: www.snre.umich.edu/ecomgt/evaluation/templates.htmPatton,
M.Q. (1987). Qualitative Research Evaluation Methods. Thousand
Oaks, CA: Sage Publishers.
Thomson, G. & Hoffman, J. (2003). Measuring the success of
EE programs. Canadian Parks and Wilderness Society.
THE NATURE OF MONITORING AND EVALUATION
Definition and Purpose
by Phil Bartle, PhD
Workshop Handout
This explains what monitoring is and the purposes it serves
What is Monitoring?Monitoring is the regular observation and
recording of activities taking place in a project or programme. It
is a process of routinely gathering information on all aspects of
the project.
To monitor is to check on how project activities are
progressing. It is observation; systematic and purposeful
observation.
Monitoring also involves giving feedback about the progress of
the project to the donors, implementors and beneficiaries of the
project.
Reporting enables the gathered information to be used in making
decisions for improving project performance.
Purpose of Monitoring:Monitoring is very important in project
planning and implementation.
It is like watching where you are going while riding a bicycle;
you can adjust as you go along and ensure that you are on the right
track.
Monitoring provides information that will be useful in:
Analysing the situation in the community and its project;
Determining whether the inputs in the project are well
utilized;
Identifying problems facing the community or project and finding
solutions;
Ensuring all activities are carried out properly by the right
people and in time;
Using lessons from one project experience on to another; and
Determining whether the way the project was planned is the most
appropriate way of solving the problem at hand.
Community Based Monitoring Workshop:
Copyright 1967, 1987, 2007 Phil BartleWeb Design by Lourdes
Sada
Last update: 2011.09.30
Definitions, Meaning & characteristics of Planning.
Meaning and Concept of PlanningIn simple words, planning is
deciding in advance what is to be done, when where, how and by whom
it is to be done. Planning bridges the gap from where we are to
where we want to go. It includes the selection of objectives,
policies, procedures and programmes from among alternatives. A plan
is a predetermined course of action to achieve a specified goal. It
is an intellectual process characterized by thinking before doing.
It is an attempt on the part of manager to anticipate the future in
order to achieve better performance. Planning is the primary
function of management.
Definitions of PlanningDifferent authors have given different
definitions of planning from time to time. The main definitions of
planning are as follows:
According to Alford and Beatt, "Planning is the thinking
process, the organized foresight, the vision based on fact and
experience that is required for intelligent action."
According to Theo Haimann, "Planning is deciding in advance what
is to be done. When a manager plans, he projects a course of action
for further attempting to achieve a consistent co-ordinate
structure of operations aimed at the desired results.
According to Billy E. Goetz, "Planning is fundamentally choosing
and a planning problem arises when an alternative course of action
is discovered."
According to Koontz and O' Donnell, "Planning is an intellectual
process, conscious determination of course of action, the basing of
decision on purpose, facts and considered estimates."
According to Allen, "A plan is a trap laid to capture the
future."
Nature / Characteristics of Planning
The main characteristics or nature of planning is given
below:
Planning is an Intellectual ProcessPlanning is an intellectual
process of thinking in advance. It is a process of deciding the
future on the series of events to follow. Planning is a process
where a number of steps are to be taken to decide the future course
of action. Managers or executives have to consider various courses
of action, achieve the desired goals, go in details of the pros and
cons of every course of action and then finally decide what course
of action may suit them best.
Planning Contributes to the ObjectivesPlanning contributes
positively in attaining the objectives of the business enterprise.
Since plans are there from the very first stage of operation, the
management is able to handle every problem successfully. Plan try
to set everything right. A purposeful, sound and effective planning
process knows how and when to tackle a problem. This leads to
success. Objectives thus are easily achieved.
Planning is a Primary Function of ManagementPlanning precedes
other functions in the management process. Certainly, setting of
goals to be achieved and lines of action to be followed precedes
the organization, direction, supervision and control. No doubt,
planning precedes other functions of management. It is primary
requisite before other managerial functions step in. But all
functions are inter-connected. It is mixed in all managerial
functions but there too it gets precedence. It thus gets primary
everywhere.
A continuous ProcessPlanning is a continuous process and a never
ending activity of a manager in an enterprise based upon some
assumptions which may or may not come true in the future.
Therefore, the manager has to go on modifying revising and
adjusting plans in the light of changing circumstances. According
to George R. Terry, "Planning is a continuous process and there is
no end to it. It involves continuous collection, evaluation and
selection of data, and scientific investigation and analysis of the
possible alternative courses of action and the selection of the
best alternative.
Planning Pervades Managerial ActivitiesFrom primary of planning
follows pervasiveness of planning. It is the function of every
managerial personnel. The character, nature and scope of planning
may change fro personnel to personnel but the planning as an action
remains intact. According to Billy E. Goetz, "Plans cannot make an
enterprise successful. Action is required, the enterprise must
operate managerial planning seeks to achieve a consistent,
coordinated structure of operations focused on desired trends.
Without plans, action must become merely activity producing nothing
but chaos."
Role, Significance, Importance & Advantages of Planning
An organisation without planning is like a sailboat minus its
rudder. Without planning, organisation, are subject to the winds of
organizational change. Planning is one of the most important and
crucial functions of management. According to Koontz and O'Donnell,
"Without planning business becomes random in nature and decisions
become meaningless and adhoc choices." According to Geroge R.
Terry, "Planning is the foundation of most successful actions of
any enterprise." Planning becomes necessary due to the following
reasons:
Reduction of UncertaintyFuture is always full of uncertainties.
A business organisation has to function in these uncertainties. It
can operate successfully if it is able to predict the
uncertainties. Some of the uncertainties can be predicted by
undertaking systematic. Some of the uncertainties can be predicted
by undertaking systematic forecasting. Thus, planning helps in
foreseeing uncertainties which may be caused by changes in
technology, fashion and taste of people, government rules and
regulations, etc.
Better Utilization of ResourcesAn important advantage of
planning is that it makes effective and proper utilization of
enterprise resources. It identifies all such available resources
and makes optimum use of these resources.
Increases Organizational EffectivenessPlanning ensures
organizational effectiveness. Effectiveness ensures that the
organisation is in a position to achieve its objective due to
increased efficiency of the organisation.
Reduces the Cost of PerformancePlanning assists in reducing the
cost of performance. It includes the selection of only one course
of action amongst the different courses of action that would yield
the best results at minimum cost. It removes hesitancy, avoids
crises and chaos, eliminates false steps and protects against
improper deviations.
Concentration on ObjectivesIt is a basic characteristic of
planning that it is related to the organizational objectives. All
the operations are planned to achieve the organizational
objectives. Planning facilitates the achievement of objectives by
focusing attention on them. It requires the clear definition of
objectives so that most appropriate alternative courses of action
are chosen.
Helps in Co-ordinationGood plans unify the interdepartmental
activity and clearly lay down the area of freedom in the
development of various sub-plans. Various departments work in
accordance with the overall plans of the organisation. Thus, there
is harmony in the organisation, and duplication of efforts and
conflict of jurisdiction are avoided.
Makes Control EffectivePlanning and control are inseparable in
the sense that unplanned action cannot be controlled because
control involves keeping activities on the predetermined course by
rectifying deviations from plans. Planning helps control by
furnishing standards of performance.
Encouragement to InnovationPlanning helps innovative and
creative thinking among the managers because many new ideas come to
the mind of a manager when he is planning. It creates a
forward-looking attitude among the managers.
Increase in Competitive StrengthEffective planning gives a
competitive edge to the enterprise over other enterprises that do
not have planning or have ineffective planning. This is because
planning may involve expansion of capacity, changes in work
methods, changes in quality, anticipation of tastes and fashions of
people and technological changes etc.
Delegation is FacilitatedA good plan always facilitates
delegation of authority in a better way to subordinates.
Steps involved in Planning
Planning is a process which embraces a number of steps to be
taken. Planning is an intellectual exercise and a conscious
determination of courses of action. Therefore, it requires courses
of action. The planning process is valid for one organisation and
for one plan, may not be valid for other organizations or for all
types of plans, because various factors that go into planning
process may differ from organisation to organisation or from plan
to plan. For example, planning process for a large organisation may
not be the same for a small organisation. However, the major steps
involved in the planning process of a major organisation or
enterprise are as follows:
Establishing objectivesThe first and primary step in planning
process is the establishment of planning objectives or goals.
Definite objectives, in fact, speak categorically about what is to
be done, where to place the initial emphasis and the things to be
accomplished by the network of policies, procedures, budgets and
programmes, the lack of which would invariably result in either
faulty or ineffective planning.
It needs mentioning in this connection that objectives must be
understandable and rational to make planning effective. Because the
major objective, in all enterprise, needs be translated into
derivative objective, accomplishment of enterprise objective needs
a concrete endeavor of all the departments.
Establishment of Planning PremisesPlanning premises are
assumptions about the future understanding of the expected
situations. These are the conditions under which planning
activities are to be undertaken. These premises may be internal or
external. Internal premises are internal variables that affect the
planning. These include organizational polices, various resources
and the ability of the organisation to withstand the environmental
pressure. External premises include all factors in task environment
like political, social technological, competitors' plans and
actions, government policies, market conditions. Both internal
factors should be considered in formulating plans. At the top level
mainly external premises are considered. As one moves downward,
internal premises gain importance.
Determining Alternative CoursesThe next logical step in planning
is to determine and evaluate alternative courses of action. It may
be mentioned that there can hardly be any occasion when there are
no alternatives. And it is most likely that alternatives properly
assessed may prove worthy and meaningful. As a matter of fact, it
is imperative that alternative courses of action must be developed
before deciding upon the exact plan.
Evaluation of AlternativesHaving sought out the available
alternatives along with their strong and weak points, planners are
required to evaluate the alternatives giving due weight-age to
various factors involved, for one alternative may appear to be most
profitable involving heavy cash outlay whereas the other less
profitable but involve least risk. Likewise, another course of
action may be found contributing significantly to the company's
long-range objectives although immediate expectations are likely to
go unfulfilled.
Evidently, evaluation of alternative is a must to arrive at a
decision. Otherwise, it would be difficult to choose the best
course of action in the perspective of company needs and resources
as well as objectives laid down.
Selecting a Course of ActionThe fifth step in planning is
selecting a course of action from among alternatives. In fact, it
is the point of decision-making-deciding upon the plan to be
adopted for accomplishing the enterprise objectives.
Formulating Derivative PlansTo make any planning process
complete the final step is to formulate derivative plans to give
effect to and support the basic plan. For example, if Indian
Airlines decide to run Jumbo Jets between Delhi an Patna,
obliviously, a number of derivative plans have to be framed to
support the decision, e.g., a staffing plan, operating plans for
fuelling, maintenance, stores purchase, etc. In other words, plans
do not accomplish themselves. They require to be broken down into
supporting plans. Each manager and department of the organisation
is to contribute to the accomplishment of the master plan on the
basis of the derivative plans.
Establishing Sequence of ActivitiesTiming an sequence of
activities are determined after formulating basic and derivative
plans, so that plans may be put into action. Timing is an essential
consideration in planning. It gives practical shape and concrete
form to the programmes. The starting and finishing times are fixed
for each piece of work, so as to indicate when the within what time
that work is to be commenced and completed. Bad timing of
programmes results in their failure. To maintain a symmetry of
performance and a smooth flow of work, the sequence of operation
shaped be arranged carefully by giving priorities to some work in
preference to others. Under sequence it should be decided as to who
will don what and at what time.
Feedback or Follow-up ActionFormulating plans and chalking out
of programmes are not sufficient, unless follow-up action is
provided to see that plans so prepared and programmes chalked out
are being carried out in accordance with the plan and to see
whether these are not kept in cold storage. It is also required to
see whether the plan is working well in the present situation. If
conditions have changed, the plan current plan has become outdated
or inoperative it should be replaced by another plan. A regular
follow-up is necessary and desirable from effective implementation
and accomplishment of tasks assigned.
The plan should be communicated to all persons concerned in the
organisation. Its objectives and course of action must be clearly
defined leaving no ambiguity in the minds of those who are
responsible for its execution. Planning is effective only when the
persons involved work in a team spirit and all are committed to the
objectives, policies, programmes, strategies envisaged in the
plan.
Dr. Michaela Zint, Project Leader
Dr. Michaela Zint, Project Leader
Dr. Michaela Zint, Project Leader