This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
How is Easy Outcomes done?• A template in DoView outcomes software is filled out *• It’s based around drawing an outcomes model (also called a program
logic, results chain, strategy map, ways-means diagram) using a set of guidelines
• Once the outcomes model has been drawn, you put onto it indicators, evaluation questions etc.
• You the analyse what’s appropriate, feasible and affordable (e.g. for outcomes evaluation you use the 7 possible types of outcome evaluation identified in Easy Outcomes)
• Once the work is done you are easily able to quickly overview the whole monitoring and evaluation planning
* If one wanted to, Easy Outcomes can be implemented using standard office software but it would take longer to do so.
• Indicators are routinely collected measures of outcomes or steps• You may or may not be collecting these indicators at the moment• Use the picture you now have of which indicators you are already
(or are planning to) collect to think about your indicator coverage• Example: Potential indicators mapped onto the outcomes model
• Put possible evaluation questions next to the outcomes or steps they apply to
• Put both outcome evaluation questions (proving that high level outcomes have been changed by the program and non-outcome evaluation questions (asking about how to improve the program or about how the process works)
• Don’t worry whether or not you can answer these evaluation questions at this stage
• Example: Evaluation questions mapped onto the outcomes model
7. Decide which evaluation questions to try to answer
• Don’t just assume that you have to, or will be able to, answer high level outcome evaluation questions. If you can that’s good, but often it would be better for your funding or controlling organization to undertake such outcome evaluation (e.g. by intensively evaluating a pilot program rather than demanding all programs have high level outcome evaluation)
• Example: Evaluation question analysis looking at which evaluation questions you’re going to attempt to answer. The evaluation question ‘Did the party make the guests happy?’ will be examined in more detail in the next step.
• There’s only a limited range of possible outcome evaluation designs and Easy Outcomes currently divides these into seven possible types
• Work out whether any of these are possible for your high level evaluation question(s) (if you don’t know how to do this, involve someone else who does) (more information: www.systematicoutcomesanalysis.org/oiiwa/model/designs.html)
• Example: Analysis of the appropriateness, feasibility and affordability of the first three possible high level outcome evaluation designs to answer the evaluation question ‘Did the party make the guests happy?’
8b. Work out what outcome evaluation is possible (cont.)
• Note that only some of the possible high level outcome evaluation designs will give you information needed in later stages of Easy Outcomes. In particular, certain types of economic analysis can only be done if there is a quantitative measures of the effect of your activity on high level outcomes (only the first four designs have the potential to provide this)
• Example: Appropriateness, feasibility and affordability of last four high level outcome evaluation designs
• Set out your evaluation projects and list under them the evaluation questions they’ll be answering
• This helps you to be very clear about exactly which evaluation question(s) are being answered by a particular evaluation project (very helpful in large scale evaluations)
• Example: Evaluation projects and the evaluation questions they will attempt to answer
• There are only a limited range of possible economic evaluation designs and Easy Outcomes currently divides these into ten possible types
• Work out whether any of these are possible (if you don’t know how to do this, involve someone else who does) (more information: www.systematicoutcomesanalysis.org/oiiwa/model/economic.html)
• Example: Two examples of types of economic analysis which could
You can chose between two major overall evaluation schemes:
1. Doing outcome evaluation on the whole program roll-out
2. Just doing outcome evaluation on a pilot and then making sure that the full program roll-out is applying best practice learnt in the pilot (but not doing outcome evaluation on the full roll-out)
Example: Shows the two possible major overall evaluation schemes for the party example
12. Work out delegated or contracted accountabilities
There are three possible types of contracting in Easy Outcomes:
1. Contracting for outputs2. Contracting for outputs and ‘Managing for outcomes’ *3. Contracting for not fully controllable outcomes *It’s very important to be clear about which kind of contracting you’re doing.
Example: The waiting staff are contracted to be accountable for indicators related to the food and drink being adequately distributed (outputs) but not further up the outcomes model
* Type 2. does not involve being accountable for high level outcomes. Type 3 occurs, for example, in the private sector when a CEOs gets a bonus related to the organization’s share price.
• Demanding only certain types of evaluation design be used (e.g. experimental outcome evaluation) regardless of the appropriateness, feasibility or affordability of these designs in particular cases
• Demanding all programs be evaluated and then getting weak evaluations of many programs rather than robust evaluations of a selected few which could be done if evaluation resources and effort were more focused
• Insisting that players ‘contract for outcomes’ or be ‘accountable for outcomes’ and exposing both contractors and contractees to risks in situations where it is difficult to actually attribute change to a
• Think of Easy Outcomes as a standard format for setting out outcomes, monitoring and evaluation planning
• This is particularly useful where a number of programs are reporting to a decision maker and means the decision maker or his or her staff do not have to grapple with many different reporting formats
• Not having a standard format for reporting on outcomes and evaluation planning is a bit like not having a standard format for reporting accounting information – likely to lead to a great deal of additional work as you have to come to terms with different frameworks
How is Easy Outcomes done?• A template in DoView outcomes software is filled out *• It’s based around drawing an outcomes model (also called a program
logic, results chain, strategy map, ways-means diagram) using a set of guidelines
• Once the outcomes model has been drawn, you put onto it indicators, evaluation questions etc.
• You do some analysis of what’s appropriate, feasible and affordable (e.g. for outcomes evaluation you use the 7 possible types of outcome evaluation identified in Easy Outcomes)
• Once the work is done you are easily able to quickly overview the whole monitoring and evaluation planning
* If one wanted to, Easy Outcomes can be implemented using standard office software but it would take longer to do so.
Since strategic advantage often lies in difficult to measure areas…
• Strategic advantage in the public, private and community sectors often lies in ‘hard to measure’ areas
• Well measured areas where causation has been established through monitoring and evaluation are usually full of players and activity
• In the private sector this makes them commodity markets, in the public & community sectors they’re the places where much activity is already being focused
…It’s important that you get better control of evaluation, monitoring and
outcomes planning
• An unsophisticated ‘go forth and evaluate’ or ‘just do outcomes evaluation’ approach will not work (you’ll end up doing the measurable, not necessarily the strategic)
• The Easy Outcomes approach means that you and your staff will know what is known, what is not yet known and what can be known to aid decision making
• If you’re going to get optimal strategic intelligence for decision making you need to make the decisions about how to deploy your limited monitoring and evaluation resources
Put the outcomes models at the heart of your organization
• The same Easy Outcomes model that has been built for monitoring and evaluation can also be used for:
– Defining your vision and mission– Priority setting in strategic planning– Identifying the outcomes of your organizational units– Organizing the evidence base around your interventions– Talking to outside stakeholders about what you’re doing– Delegating and contracting
The process• Draw an outcomes model• Set strategic priorities for action• Put indicators onto your model• List indicator projects• Put evaluation questions onto your model• Decide which evaluation questions to try to answer• Work out what outcome evaluation is possible• List evaluation projects• Work out what economic evaluation to do• Work out overall evaluation scheme• Work out delegated or contracting accountabilities
The guidelines1. Use outcomes not activities2. Draw the ‘cascading set of causes in the real world’3. Don’t force models into horizontal levels4. Don’t make ‘siloed’ outcomes models5. Singular not composite steps and outcomes6. Keep outcomes short7. Put outcomes into an hierarchical order8. Include all relevant steps needed to cause an outcome9. Keep measurement separate from steps and outcomes10. Put a ‘value’ in front of steps and outcomes11. Make just the right number of ‘slices’ 12. An organization does not necessarily need a single high level
outcome13. Include both current and future priorities
2. Cascading set of causes in the real worldDraw all of the steps (causes) leading to high level outcomes in the outside world. Drill down as far as you like. Draw the full outcomes model as on the left, don’t limit yourself just to measurable outcomes (middle model) or those that it can be proved have been changed by a player (attributable outcomes – as in the right hand model where only outcomes which it can be proved were caused by a player have been included in the model).
3. Don’t force models into horizontal levelsDifferent outcomes models will have more or less steps at a particular level, don’t force steps into levels or restrict the number at a particular level (as has been done in these diagrams). Outputs can be identified on your model but they don’t have to lie at a particular level.
In the ‘siloed’ model on the left, lower level steps are only allowed to cause a single high level outcome. In the real world, a lower level step may well cause more than one high level outcome (as in the model on the right shows where the blue step causes both the blue and the yellow higher level outcome).
Make sure that outcomes are not ‘composite’ like the one on the left hand side which contains both an ‘end’ (improve father-son relationships) and a ‘means’ (teaching them to fish together). Composite outcomes lock you permanently into using one strategy to get an outcome. If by, through, to, in order to or similar terms appear in an outcome they show that it is a composite outcome. Divide the composite outcome into two or more outcomes and open up the possibility that there is another way (strategy) of getting to the outcome (e.g. the blue box in the diagram on the right).
Keep outcomes and steps short, as in the one on the right rather than the one on the left. If you need to, keep an additional longer description of the outcome separately. For every additional word you use in an outcome name, you will lose or confuse some readers.
The outcomes on the left are not in an hierarchical order. The ones on the right are. You can tell that there is an hierarchical order to the outcomes on the right because if you could magically achieve the top one, you would not bother intervening to do the lower ones.
8. Include all relevant steps needed to cause an outcome or step
The model on the left is incomplete because it does not include an essential step (the red one in the model on the right) which is needed to cause the outcome above to occur. Include all notable steps needed. You need to use your common sense, because there will be steps which are needed but which can be taken for granted in some situations (e.g. electricity, telephone systems) you don’t need to include such ‘taken for granted’steps where it can be assumed they will be present.
9. Keep measurement separate from steps & outcomesFirst draw your outcomes model without worrying about whether you can measure the steps and outcomes (as in the model on the left). Then insert measurements as indicators (as in the model on the right). If you put measurements into your actual outcomes model you cannot deal with the fact that particular ways of measuring an outcome may be inaccurate. In a few cases, where measurement causes something to happen (e.g. audit processes), they may also appear as steps in their own right.
Put a ‘value’ such as successful, sufficient, effective or accurate in front of your outcomes and steps. You don't need to define this when you are first building your outcomes model. By putting this value in you often limit long discussions about the exact amount of the step or outcome that is needed. If you take too much time up with such discussions when first drawing your model in a workshop situation you’re likely to run out of time). The question of how much is sufficient etc can then become the subject of an evaluation project if necessary.
You model should be made up of just the right number of ‘slices’ needed to represent the world of outcomes and steps in which you are intervening. The actual world of causes is a multi-dimensional world which you need to somehow represent in a finite set of dimensions. Your challenge is to ‘slice’ that multi-dimensional world up into a set of slices which make sense to you and your stakeholders. For instance below you can see a set of three slices (there would be a set of outcomes below each of these). Each one is based on a different conceptual level: national, locality and individual level. It may take some time to work out the best set of slices to use in a particular model. Just try developing different slices as they occur to you and then as you progress in drawing your model, get rid of duplicate steps and outcomes and join or separate them as you think appropriate.
12. An organization does not necessarily need a single high level outcome
Where an organization has been delegated or contracted to intervene in more than one area, it’s appropriate for it to have separate outcomes models for each area. On the left below is an artificial attempt at a single outcomes model. Having separate outcomes models for each area (as on the right) means that the organization is in a better position to collaborate with outside organizations working in the same area. For instance, the two green outcomes on the extreme right (which are the focus of another organization) can be much more easily integrated into the outcomes model on this topic when it has been set up as in the right hand rather than the left hand model.
13. Include both current and future prioritiesYour outcomes models should include all relevant causes in the real world and should not be limited just to your current priorities (determined by resourcing or other constraints). The model on the right is limited to only priorities and so does not give a full strategic view of the world in which you’re intervening. Highlighting priorities onto a full outcomes model (as in the middle example) is a better approach and works well in the public sector where public sector employees can give ‘free and frank’ advice about the world as it is and elected officials can then determine current priorities on this model.