WELCOME June 2012 Evaluation Planning and Management John Wooten ~Lynn Keeys.

WELCOME

June 2012

Evaluation Planning and Management

John Wooten ~ Lynn Keeys

Session 8: Evaluation Team Debriefing and ReportObjectives:• Review some important considerations in debriefing

evaluation teams• Review requirements of core sections of the evaluation

report• Understand how to transition smoothly from team

debriefings to the draft and final evaluation reports

Evaluation Road Map :Evaluation Stage 5

2

Evaluation Team Debriefing and Report

• Don’t engage in detail meddling• Don’t disallow team’s structuring inputs• Do consider iterative debriefings• Do understand how report drafting is unfolding • Do be prepared to guide the team in developing an

acceptable product

Some DOs and DON’Ts to remember…

3

Evaluation Team Debriefing and ReportWhich comes first? Why?

Draft evaluation report?

Team debriefing?

4

Evaluation Team Debriefing and Report

Draft evaluation report?

Team debriefing?

Consider requesting an annotated report outline

before the debriefing

Which comes first? Why?

Tip:

5

Evaluation Team Debriefing

• Finalize/activate team debriefing plan• Iterative debriefings, successively higher level• Representatively participatory/transparent• Evaluation team inputs on structure• ~One hour (20 min. presentation)

• Organize around either :- Findings, conclusions, recommendations, lessons- Key evaluation questions - Project components

• Agree on modifications for next debriefing

Evaluation team debriefing

6

Evaluation Team Debriefing

• Provide team brief, written summary of decision points, guidance, OR

• Request annotated outline of draft report (highlight decision points, guidance from debriefings)

• Promptly ID in writing misunderstandings, internal conflicts, differences of opinion; resolve as indicated

• Preview all data to be submitted in agreed formats• Identify data-related issues in advance

Tip: Avoid making assumptions

Evaluation team debriefing … FINAL

7

Evaluation Team ReportAll conspiring to answer…

Whether /why intended results were/were not

achieved

Accurate description

Existing,

new info (evidence)

Analysis, interpretation of

facts

General conclusions

Set of expert recs.

8

Evaluation Team ReportDraft evaluation report…

AIMED AT GENERAL PROFESSIONAL

READER

IN TARGETED AUDIENCES

CLEAR

CONCISE

CREDIBLE

IMPACTFUL

PERSUASIVE

9

Evaluation Team Report

• Final evaluation SOW (illustrative outline)

• Evaluation work plan (finalized/approved)

• Cursory review field experience (final +)

• Judicious use of annotated outline (R.e.a.l.l.y. FINAL)

So when is final FINAL?

Draft evaluation report…Iterative outline

10

Evaluation Team Report

1. USAID’s Monitoring and Evaluation TIPS #17,

“Constructing and Evaluation Report” (Table 1, p. 8 guided outline)

2. Un-numbered TIPS (same title)

(Annotated report outline; Management Systems Int’l, 4/2006)

3. “Checklist for Assessing USAID Evaluation Reports”

(Hard copies @ Tab 7)

Draft evaluation report…References

11

Evaluation Team Report

RecommendationsProposed actions for management

ConclusionsInterpretations and judgments

based on the findings

FindingsEmpirical facts collected during the

evaluationArtfully constructed, to convey/reflect team’s clarity, skill, professional judgments,

experience and passion, contributing to decision-making processes that lie ahead

Organizing key elements of the evaluation report

Fig. 1, TIPS #17, p. 2

Attribution?

Counterfactual?

12

Evaluation Team Report

• Organize around either :- Findings, conclusions, recommendations, lessons- Key evaluation questions - Project components- Otherwise

• Share your own perspectives

• Share good, bad, ugly samples/experiences• Understand key sectional differences…

Draft evaluation report…Organization

13

Tips on Findings?

Evaluation Team Report

• Based on analysis: - Facts -- empirical evidence - Typically emerge from data analysis- Begin to answer key evaluation questions

Draft evaluation report…Findings

14

Evaluation Team Report

• Bundle findings that answer a macro-question• Examine how management transformed inputs-to-outputs• Examine how management “positioned” outputs to

influence (+,-) outcomes (causality)

• Relate findings to project design (theory of change, results framework, logical framework, critical assumptions)

• Include clear, hard-hitting summary of data analyses

Draft evaluation report…Findings Tips

15

Evaluation Team Report

• Based on judgments: - Involve deductive judgments (meanings)- Emerge when applying reasoning to interpret evaluation

findings

Draft evaluation report…Conclusions

Tips on Conclusions?

16

Evaluation Team Report

• Communicate importance, benefit, impact of empirical findings about project, context, problem, management framework

• Demonstrate understanding of context• Summarize team’s views on whether/why a project

succeeded or failed achieve various design aspects• Clearly answer “So what?” questions• Clearly/logically support empirical findings• Reflect skills, expertise of team members

Draft evaluation report…Conclusions Tips

17

Evaluation Team Report

• Based on reflective experiences: - Answer, “So what do we do about it?” - Are judgments on what needs adjusting to enhance likely

achievement of results in the current or future project or related programs

Draft evaluation report…Recommendations

Tips on Recommendations?

18

Evaluation Team Report

• Flow clearly/logically from findings/conclusions• Anticipate, respond to pending decisions cited in SOW• Are actionable (i.e., feasible, cite responsible agency or authority,

within their ‘manageable interests’)

Draft evaluation report…Recommendations Tips

But how to write the report?!? 19

…by report outline?

…by technical specialty?

Consider how to avoid a report that reads like…

…a committee product?!

> BONUS TIP <

But how to write the report?!?

Draft evaluation report…Writing

Evaluation Team Report

20

Evaluation Team Report

• Triangulate review of draft report based on:- Final evaluation SOW- Written agreements from debriefings - Evaluation report checklist

• Organize peer reviews (w/strict time limit)

• Allow ample time for stakeholders’ review

(cultural, language, availability constraints)

• Push to resolve ALL dissenting perspectives, and feedback conflicts (…and if not resolved?)

Draft evaluation report…Review

21

Evaluation Team ReportDraft evaluation report…Review

22

• Prepare timely, written, contradictions-free feedback (the “corporate” perspective ~ USAID et al)

• Follow-up w/confirmation phone call/meeting• Limit track changes to minor editorials only; provide major,

substantive comments separately• Re-confirm timeframe for receipt of final report• Reconfirm LOE/budget availability

Evaluation Team Report

• Review per evaluation checklist• Verify responsiveness to the “corporate” feedback• Negotiate required refinements• Review, confirm content/completeness of all quantitative

data in agreed formats • Verify all SOW terms are fully satisfied• Issue acceptance letter (per A&A Office guidance)

Final evaluation report…Review/Acceptance

Congrats on this significant milestone~

23

([email protected]) ([email protected])

24

Evaluation Planning and Management

John Wooten ~ Lynn Keeys

Thank you~